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ABSTRACT 

The Bilingual Education Evaluation System was 
developed to help local bilingual education projects overcome 
evaluation obstacles and design sound, useful evaluations within the 
constraints of available funding and with responsiveness to current 
federal legislation and regulations. The evaluation system proposed 
involves a proces:^ ::omponent, an outcome component, and procedures 
for integrating them. An innovative element is a gap-reduction design 
for assessing student outcomes. This element assesses the academic 
growth of project participants relative to one of two recommended 
comparison groups: national norms or non-project students in the same 
grade. The user's guide to the evaluation system consists of two 
volumes. The first volume contains all of the system's procedures and 
practices. The system is divided into nine steps, each corresponding 
to a chapter of this volume: assuring that the project is evaluable; 
planning the evaluation; documenting program processes; 
selecting/adapting/developing instruments for assessing student 
outcomes; collecting outcome data; implementing the gap-reduction 
design; processing and analyzing data; integrating and interpreting 
result '^; and preparing evaluation reports. The second volume contains 
explanations and discussions of the rationale underlying 
recommendations for procedures and practices made in the first 
volume, and detailed guidelines on how to perform certain tasks. 
Topics include the following: classroom observation; interrater 
reliability; functional level testing; translating tests into other 
languages; major publishers of nationally standardized achievement 
tests; test reliability; quasi-experimental designs; gap reduction 
calculations; extrapolation procedures; and correcting for 
regression. (MSE) 
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1. INTRODUCTION 



Title VII evaluations 
have been required 
since 1968. 



Since the Bilingual Education Act (Title VII of the 
Elementary and Secondary Education Act) was 
passed in 1968, thousands of projects have received 
Federal funds to help them address the needs of 
limited-English-proficient students-students whose 
native languages and cultural backgrounds separate 
them from mainstream students. 



Projects funded under the original Act were required 
to conduct self-evaluations, and some form of evalua- 
tion requirement has been in force-almost without 
interruption-from 1968 to the present time. 



Most early evaluations 
were seriously flawed 



As a result of the legislation and associated regula- 
tions, a very large number of evaluation reports have 
been produced with Title Vn funds. Unfortunately, 
most of these evaluations have not proven very useful 
either in helping local projects to demonstrate their 
effectiveness or in helping them to improve their 
services. Some of the reasons for these problems 
have been: 



• lack of evaluation expertise at the local level. 

• inadequate guidelines for evaluation. 

• insufficient technical assistance to local projects. 

• limited availability of funds. 
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What led to development 

of the BEES? The Bilingual Education Evaluation System (BEES) 

described in this Users' Guide was developed to help 
local projects overcome these problems and design 
sound, useful evaluations within the constraints of 
available fiirding. Its development was sponsored by 
the Department of Education, which also established 
two regional Evaluation Assistance Centers (EACs) 
to help local projects improve their evaluations. 

Design Objectives for the BEES 

What objectives were 

established for the BEES? Development of the BEES was guided by three 

primary design objectives: 

9 The system should reflect the sum total of 
knowledge gained from previous work in bilin- 
gual education evaluation. To meet this objec- 
tive, the Guide's authors conducted a literature 
review which is summarized in a separate 
document.^ 

# The system should be useful at the local level for 
purposes of project improvement. To meet this 
objective, evaluation has been made an integral 
part of the project development process-from 



1. Tallmadge, G. K., Lam, T. C. M., & Gamel, N. N. (1987). The evaluation of bilin- 
gual education programs for language-minority lirnited-English-proficient students: A 
status report with recommendations for future development. Mountain View, CA: 
RMC Research Corporation. 
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needs assessment, through project planning and 
implementation, to project modification. 



9 The system should be totally responsive to the 
current federal legislation and regulations 
governing the evaluation of Title VII projects. 



Nature of the BEES 



What is encompassed 

by the BEES? The BEES is a total evaluation system that involves a 

process component, an outcome component, and 
procedures for integrating the two. The most innova- 
tive element of the system, however, is the gap- 
reduction design that is recommended for assessing 
student outcomes. 



The gap-reduction design 
is recommended for 

assessing outcomes. In developing the BEES we assumed that most 

projects would* find it difficult or impossible to 
implement a traditional true or quasi-experimental 
design* For this reason, we formulated a design (the 
gap-reduction design) that is easy to implement, 
satisfies the regulations' requirements, and does not 
require a nonproject comparison group made up of 
students similar to chose served by the project. 

The gap-reduction design assesses the academic 
growth of project participants relative to one of two 
recommended comparison groups-national norms or 
the nonproject grade mates of the project students. 
These comparisons provide useful information about 
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Content and organization 
of the Users' Guide. 



how well the students are progressing-but v;ithout 
yielding a quantitative estimate of the project's im- 
pact. The evaiuator musi rely on a wide array of in- 
formation and findings about both processes and 
outcomes to form judgments about project eftective- 
ness. 

The BEES emphasizes quality control and explicit 
acknowledgment of unavoidable problems as 
methods of avoiding erroneous conclusions. 

Overview of the Users' Guide 

The Users' Guide is organized into Uvo volumes. This 
first volume contains all the system's evaluation pro- 
cedures and practices. Volume n contains explana- 
tions and discussions of the rationales underlying 
various recommendations, as well as detailed 
guidelines on how to perform certain tasks described 
in Volume I. 

The Bilingual Education Evaluation System includes 
nine major steps, each of which corresponds to a 
chapter of Volume I of the Users' Guide. 

o Assuring that the Project is Evaluable (Chapter 
II). 

o Planning the Evaluation (Chapter HI). 

o Documenting Program Processes (Chapter IV). 

• Selecting/Adapting/Developing Instruments for 
Assessing Student Outcomes (Chapter V). 

• Collecting Outcome Data (Chapter VI). 
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• Implementing the Gap-Reduction Design 
(Chapter VH). 

• Processing and Analyzing Data (Chapter VIII). 

e Integrating and Interpreting Results (Chapter 
IX). 

% Preparing Evaluation Reports (Chapter X). 
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n. ASSURING THE PROJECTS EVALUABILHY 



"Would you tell me, please, which way I ought to go from here?" 
"That depends a good d(^ on where you want to get to," said the Cat. 
"I don't much care where-" said Alice. 
"Then it doesn't matter which way you go," said the Cat. 

(Alice in Wonderland) 



The preceding quotation is directly relevant to the 
evaluability of projects. To paraphrase, if you don't 
know what a project is trying to accomplish-and why 
and how-you can't determine how successful it is, or 
how to improve it. 

Evaluability implies the potential for meaningfiil and 
accurate evaluation which can lead to valid conclu- 
sions about project effectiveness and to sound 
recommendations for project improvement. A 
project is evaluable if its rationale, design, and ex- 
pected outcomes are clearly defined ^nd logically 
consistent. Assuring the project's evaluability is the 
first step in the evaluation process. 

Evaluability assurance should 

be a collaborative endeavor. Assming evaluability and planning for the evaluation 

should be collaborative efforts involving the project 
director, the evaluator, and (if possible) the teachmg 
staff. This group should review needs assessment 
data, descriptions of project goals and objectives, and 



What is evaluability? 
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planned project activities while addressing the 
following questions: 

Four evaluability issuns. e Is the needs assessment adequate? 

9 Are the project goals and objectives adequately 

formulated and appropriate to needs? 
e Is the project design adequately described and 

consistent with goals and objectives? 
• Are the evaluation questions adequately 

defined? 



A flow diagram representing the process of assuring 
the evaluability of a project is displayed in Figure 1. 
In the remainder of this chapter, we provide some 
guidelines for addressing the four issues to be con- 
sidered in assuring evaluability. 

Is the Needs Assessment Adequate? 

What constitutes an 

adequate needs assessment? Needs assessment is the process of: 



• identifying the academic achievement and other 
school-related deficits of the target students. 

o establishing a baselme against which to compare 
students' performance after they have received 
project services. 



The current (1986) regulations specify that achieve- 
ment must be assessed in English language 
proficiency. That term is used broadly to denote 
proficiency in speaking, reading, writing, and/or un- 
derstanding English. Proficiency must also be 
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Project Design 



Assess 
Student 
Needs 



Establish Project 
Goals and 
Objectives 



Plan 
(Design) 
Project 



Develop 
Evaluation 
Questions 



Is the Needs 
Assessment 
Adequate? 



Yes ^ 



Are the Goals and 

Objectives 
Adequately Formu- 
lated and Appro- 
priate to Needs? 



— 



1 



No 



Impossible 
to Assess the 
Appropiateness 
offtoject 
Objectives 




1 



Is the Project Design 

Adequately 
Described and Corsis- 
tent with the Goak 
and Objectives? 



Yes 



I 




i 



Are the 
Evaluation Ques- 
tions Adequately 
Defined? 



I 



.J No 



I 



I I 
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Hgure 1. Interaction of project design and evaluabUity assurance. 
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assessed in the students' native language (for projects 
of developmental bilingual education) and other 
courses or subjects of study. Reliable and objective 
assessment methods must be used in all of these 
assessments according to the current bilingual educa- 
tion regulations [§50L31(a)(l&2)]. 

English language 

proficiency measures. For the assessment of needs in English language 

proficiency, it would be desirable to have as many of 
the following indicators as possible and/or 
appropriate: 



© home language survey, 
e English language proficiency test. 
e standardized English reading readiness or read- 
ing test. 
© teacher evaluations. 



Native language 
proficiency measures. 



For the assessment of native-language proficiency, it 
would be desirable to have as many of the following 
indicators as possible and/or appropriate: 



• home language survey. 

© native-language proficiency test (if available). 

© teacher/aide evaluations. 

© records of prior schooling. 



Academic subject 
matter measures. 



For the assessment of subject matter knowledge in 
other academic content areas, it would be desirable 




to have as many of the following indicators as 
possible: 



Different students may 
have different needs. 



® standardized achievement test in written English, 

e standardized achievement test written in LI. 

9 Other achievement tests in LI. 

0 records of prior schooling. 



If the target students constitute a homogeneous 
group, needs may be appropriately characterized 
using means, medians, or other descriptive statistics. 
On the other hand, where the group to be served is 
heterogeneous in terms, for example, of ethnic 
origin, literacy in LI, or amount of prior schooling, it 
will be appropriate to make a careful note of the dif- 
ferent needs of different subgroups. 



If the already collected needs assessment data are 
not complete or adequate^ you must work out a 
strategy for collecting and documenting adequate in- 
formation about student needs. 

Are Project Goals and Objectives Adequately 
Formulated? 



Project objectives must 
relate to student needs. 



Outcome goals for projects may be no more than re- 
statements of identified student needs. If, for e^:- 
ample, the needs assessment found that the target 
LEP students were two years behind their 
mainstream peers in some academic area, the cor- 
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responding project goal would be to reduce or 
eliminate that achievement gap. 



The primary purpose of adequately formulated goals 
and objectives is to facilitate design of the interven- 
tion that will address the identified needs. A goal as 
broad as reducing the English language proficiency 
gap is not, however, particularly useful for that pur- 
pose. Broadly stated goals need to be broken dovra 
in^o their component objectives. To illustrate this 
point, the goal of gap reduction might be broken 
down into objectives such as: 

• developing literaq^ skills in LI. 

• transferring LI literacy sidlls to English. 

© enhancing the English vocabularies of target 
students. 

Each of these objectives could profitably be broken 
down into still more specific objectives. Literacy 
skills in LI (or English) might include, for example, 
recognizing the letters of the alphabet, knowing 
letter-sound correspondences, being able to decode 
letter combinations and words, etc. The greater the 
level of detail, the more useful the objectives will be 
in the project design process. The one essential in- 
gredient of a well formulated objective is that it re- 
lates clearly to an identified student need. Going 
back to our example, while some educators might 
debate the effectiveness of teaching decoding skills, 
those skills are clearly relevant. 



Goals and objectives 
should determine project 
design. 
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All objectives must be stated in measurable terms so 
that you can subsequently determine whether they 
have been achieved. "Enhancing students' apprecia- 
tion of music" is an example of an inadequately 
stated objective since appreciation of music can 
mean many different things to different people. If, 
on the other hand, you were to say, "music apprecia- 
tion as measured by such and such a test," you would 
have solved the problem by making the objective 
measurable. Some people might not agree that the 
test really measured music appreciation, but it would 
at least be clear what you hoped to impart to the 
students. 

Stating objectives in measurable terms does not 
imply that you must establish a quantitative level of 
performance as a criterion of success. Keeping up 
with, or catching up to, mainstream students will 
serve quite well as a criterion for success. There is 
no need to specify that the gap between the two 
groups' performance levels should be reduced by a 
specific number of units. In fact, establishing that 
kind of criterion is inappropriate given the state of 
our current knowledge about bilingual education. 
We simply do not know what would be reasonable to 
expect. 

The linking of objectives 

to needs. When reviewing project objectives, you should always 

verify that each one is logically linked to an identified 
student need. If it is not, then the objective should 
be discarded. 
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Objectives must be 
measurable. 



Statements of objectives for bilingual projects should 
also reflect whatever diversity of needs exists within 
the target group. With heterogeneous groups, dif- 
ferent objectives must be developed for each sub- 
group that has unique needs. 

It is also necessary for evaluability assurance pur- 
poses that separate objectives be developed for each 
year of a project. Otherwise, the required annual 
evaluations might show failure to achieve an objec- 
tive that was not relevant to that year's instructions. 
If, for example, instruction in English reading is not 
introduced until the second project year (after 
literaqr in Ll has been achieved), it would be inap- 
propriate to include gains in English reading 
achievement as a first-year objective. 

If the project objectives do not meet the cr?:eria dis- 
cussed above, they must be modified or reformu- 
lated. Without sound and well written goals and ob- 
jectives, you will not be able to assess the ap- 
propriateness of the project plan. 

Is the Project Design Adequately Described 
and Consistent with Objectives? 

All project features 
should be described 

in detail. There must be a detailed and complete description 

of the project before a useful evaluation can be un- 
dertaken. That description should encompass every 
topic to be covered in the curriculum, the classroom 



Multi-year projects require 
annual objectives. 
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Design features should 
be linked to project 
objectives. 



hours that should be devoted to it, and the instruc- 
tional practices that should be followed in teaching it. 
In addition, anticipated activities related to teacher 
training, curriculum development, and parent in- 
volvement must be described. 



The first thing to consider when evaluating a 
project's design is whether its linkages to the stated 
outcome objectives are adequately supported by logic 
or empirical evidence. In essence, you need to 
determine that the curriculum, teaching strategies, 
and other project characteristics "make pedagogic 
sense" given the project's objectives. To do so, you 
should draw upon the research literature (on effec- 
tive instructional practices, second language acquisi- 
tion, linguistics, etc.) and your own experience and 
expertise. 

If you find that some curriculum component, teach- 
ing strategy, or other project characteristic is unre- 
lated to one or more of the stated objectives, that 
project feature should be deleted and perhaps re- 
placed with a more relevant one. 

Not only must project features be clearly relevant to 
outcome objectives, they must be described in suffi- 
cient detail so that classroom observation can sub- 
sequently determine the extent to which they are 
present in the project as implemented. 
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Key features must be 

identified. The project design should spell out what should be 

going on in the classroom at any given time in terms 
of both content and method of presentation. This in- 
formation is essential so that the observer knows 
what he or she should look for during the observation 
period and will be able to recognize discrepancies 
between what was intended and what is actually hap- 
pening. Features should also be prioritized so that 
the observer's attention can be appropriately 
focused. 



Design features become 

process objectives. Features discussed in the project design^-particularly 

key features-are traditionally called process objec- 
tives. The task of determining the extent to which 
they are present in the project as implemented r 
traditionally called process evaluation. 

Each item in a detailed project description can be 
regarded as one process objective. For example, the 
statement that kindergarten smdents will receive in- 
struction in LI on the concepts of up-down, left-right, 
long-short, near-far, and inside-outside, for three 
hoi^.rs a week using the XYZ science series is a 
description of how the project is int^.nded to operate. 
Hence it is a process objective. 

If the project plan is not adequately described, you 
must work to revise it so that there is no confusion 
about how the project should operate. Chapter IV 
presents some guidelines for project description and 
documentation. 
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Are the Evaluation Questions Adequately 
Defined? 



Evaluation questions 
should derive from 
project objectives. 



Evaluation questions should be a natural outgrowth 
of your project's outcome and process objectives. In 
fact, each objective can be turned into an evaluation 
question simply by asking the extent to which it was 
achieved. If the outcome objective was to reduce the 
gap between project and mainstream students in 
reading achievement (as measured by a particular 
test), an appropriate outcome-evaluation question 
would be, "How much was the gap reduced?" If the 
process objective wa3 for teachers to conduct all in- 
struction in English, an appropriate process- 
evaluation question would be, "What percentage of 
the instruction was conducted in English?" 



Additional questions will 
derive from the regulations. 



In addition to the evaluation questions that derive 
directly from process and outcome objectives, you 
will have to consider questions posed by the current 
evaluation regulations whether or not thr relate to 
project objectives. For example, your project might 
not have objectives related to student absenteeism. 
Nevertheless, because you are required to report 
data on absenteeism, your evaluation must address 
the question, "Has the project affected the rate of 
student absenteeism?" 
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Prioritize your 
evaluation 

questions. Because of resource and budget limitations, you may 

not be able to answer every evaluation question you 
would like to ask about your project. For this reason, 
it is important that you prioritize your questions 
giving precedence to those that derive directly from 
the regulations and those that relate to key project 
features. 



In addition tQ-process- and outcome-related ques- 
tions, you should always address questions that per- 
tain to data validity. Even if you can't afford to col- 
lect or analyze data related to the validity questions, 
you should think about them throughout the evalua- 
tion period and acknowledge possible problems when 
interpreting your evaluation findings. 

Once the entire process depicted in Figure 1 has 
been successfully completed, the project's 
evaluability will have been assured, and planning for 
specific evaluation activities can begin. Evaluation 
planning is the topic of the next chapter. 
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m. PLANNING THE EVALUATION 



Plan carefully to 
avoid problems. 



Evaluation plannmg is the process of developing an 
orderly and efficient blueprint for collecting, process- 
ing,- analyzing, interpreting, and reporting the data 
needed to address the evaluation questions that have 
abready been developed. An evaluation that is not 
carefully planned is almost certain to encounter 
problems that proper planning can avoid. Among 
other things, proper planning can ensure that: 



e all the needed data are collected. 

© no unnecessary data are collected. 

9 the data are as free from errors as they can be. 

® project staff and school personnel will cooperate 

with the evaluation. 
© results are useful to project staff and available 

early enough to be helpful. 



First, study the 
Users' Guide. 



Then, start planning 
as early as possible. 



Before you begin planning your evaluation, you 
should thoroughly familiarize yourself with the con- 
tents of the Users' Guide and make sure that the 
project's evaluability has been verified as described 
in Chapter U. 



Planning should begin as soon as possible after grant 
award. An even earlier start would be highly 
desirable since the time available to complete a few 
of the tasks (e.g., selecting, ordering, and obtaining 
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Include persons with 
diverse skills. 



standardized tests) is barely adequate even with the 
six-month preservice period that is now an integral 
part of Title Vn basic grants. 

Ideally, all evaluation planning activities should be 
completed during the six-month preservice period. If 
that is not possible, you must at least complete those 
planning tasks associated with data collection smce 
actual data collection should begin at the same time 
as service delivery. 

Form an Evaluation Team 

To assure the best possible planning, you should set 
up an evaluation team that includes at least the 
project director, the project evaluator, and one or 
(preferably) more project teachers. While the 
project director is the appropriate leader, the size 
and makeup of the evaluation team may vary as a 
function of the size of your project, district resources, 
and other considerations. It is important, however, 
that all of the following competencies be 
represented: 

• methodological expertise in evaluation, 
measurement, data management, and statistical 
analysis. 

0 knowledge of bilingual education theory and 
practice. 

9 knowledge of the project's curriculum and in- 
stractional strategy. 

• interpersonal and communication skills. 
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Form action committees. The project director should organize the evaluation 

team into committees with specijSc responsibilities 
for planning and carrying out all of the required 
tasks. Initially, there should be committees for (a) 
selecting tests, (b) developing strategies and in- 
strumentation for collecting process evaluation data, 
and (c) developing procedures and instrumentation 
for documenting the project's student, treatment, and 
setting characteristics. 



Involve teachers in 
selecting tests. 



Have project designers 
plan the process 
evaluation. 



Include a person 
who knows the 
record-keeping systems. 



The test-selection committee should be made up of 
teachers and/or curriculum developers who have 
detailed knowledge of the project's specific instruc- 
tional objectives. K a person with psychometric train- 
ing is available, he or she would be a valuable mem- 
ber of the committee-especially during the test- 
selection process. 



The process-evaluation committee should include the 
project director, the project evaluator, and anyone 



else who was involved in designing the project and 
has first-hand knowledge of how the project — '"^ 
tended to be implemented. 



; was m- 



The documentation committee should include one or 
more persons familiar with all record-keeping prac- 
tices and procedures from the classroom to the dis- 
trict level. 
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Members of each committee should study those parts 
of the Users' Guide that are relevant to their respon- 
sibilities before any evaluation planning is begun. 
Then, as new planning or implementation tasks are 
undertaken, new committees should be formed, and 
additional review of Users' Guide recommendations 
should be conducted. 

Develop a calendar of 

evaluation activities. The first meeting of the evaluation-planning team 

shoT'ld be devoted to developing a calendar of events 
associated with planning for and conducting the 
evaluation. Figure 2 presents a sample format for 
such a calendar. You should use it as a starting point 
for developing your own. Whatever format you ul- 
timately decide upon, your evaluation plan should 
encompass all of the following activities: 

o selecting, modifying, and/or developing outcome 

evaluation instruments. 
6 developing instruments for collecting data on 

student, project, and setting characteristics. 
9 training data collectors and collecting data, 
e scoring and coding data. 
© setting up a data management system. 
0 creating and editing data files. 

• analyzing data. 

• interpreting analytic findings. 

e preparing the evaluation report. 

All of these topics are covered in considerable detail 
in the following chapters of this Users' Guide. 
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A^ctivitv 


Resp. 
Staff 


Start 
Date 


End 
Date 


Com- 
pleted? 


Remarks 


Outcome Measures 
Review Candidate Instruments 
Select and Order Tests 
Train Test Administrators 
Administer Pretest 
Administer Posttest 

Records Extraction 
Develop Instrument(s) 
Train Data Extractors 
Extract Data 

Teacher Reports 
Develop Instrument(s) 
Train Interviewers 
Conduct Interviews 

Classroom Observation 
Develop Instrument(s) 
Train Observers 
Conduct Observations 

Data Processing 
Develop Coding Procedures 
Train Coders 
Code Data 

Exercise Quality Control 

Data Analysis 
DevelopData Analysis Plan 
Set Up Data Management 
System 

Create Computer Files 
Edit Files 
Conduct Analysis 

Kepon rreparation 
Prepare Draft Report 
Obtain Reviewer Comments 
Revise Report 
Submit Report to Interested 
Audiences 













Figure 2. Sample chart for scheduling an evaluation. 
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Assign specific tasks 

to team members. In addition to working out a detailed schedule for 

evaluation activities, assigning responsibilities for 
completing those activities is an important part of 
the planning process. You should note that a 
column for recording personnel assignments is in- 
cluded in the sample schedule shown in Figure 2. 



Many, if not most, evaluation activities will be per- 
formed by the project director, the project 
evaluator, and project teachers. Still others may be 
performed by aides, parents, and district ad- 
ministrative and clerical staff. If possible, all of 
these individuals should participate in at least those 
planning sessions where their possible roles are 
discussed. 



Involvement breeds 

cooperation. Involving as many people as possible in planning 

the evaluation--particularly teachers and 
administrators-will be well worth whatever extra 
effort it entails. Involvement leads to a sense of 
ownership, and ownership leads to tolerance and 
cooperation when evaluation activities become 
intrusive (as they invariably do). Table 1 lists some 
of the tasks in which various members of the 
evaluation team should participate. 

Additional Planning Activities 

Good planning should not stop with the mechanics 
of setting up schedules, forming committees, and 
making personnel assignments. 
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Table 1 

Suggested Tasks for Members of the Evaluation Team 



Project Directors 

9 assuring the project's evaluability 

• reviewing the evalu^^tion plan 

• assigning tasks to members of the evaluation team 

• coordinating the evaluation team 

• selecting, modifying, or developing instruments 

• arranging and monitoring data collection and processing 

• ensuring that evaluation tasks are completed on time 

• conductiiig classroom observations 

• assisting in interpreting results 

e reviewing, editing, and writing parts of the report 

Evaluators 

• assuring the project's evaluability 

• developing or refining the evaluation design 

• selecting, modifying, or developing instruments 

• conducting classroom observations 

• managing and analyzing the data 

• interpreting the data 

• writing the evaluation report 

• presenting evaluation findings 

Teachers and Paraprofessionals 

o assisting in assuring the project's evaluability 

o reviewing the evaluation plan 

• selecting, modifying, or developing instruments 
a collecting and processing data 

0 assisting in interpreting the data 

Parents and Community Members 

• reviewing the evaluation plan 

© suggesting evaluation quv^stions 

School District Administrators 

• reviewing the evaluation plan 

o suggesting evaluation questions 

• visiting classes informally 

District Evaluation and Testing Staff 

• reviewing the evaluation plan 
« providing district data 

• assisting in the construction of the project's data base 

• reviewing draft reports 
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Overcoming resistance 
to evaluation. 



Good planning also includes: (a) smoothing the 
way for activities that could otherwise encounter 
resistance and (b) anticipating potential problems 
(and at least think'- g about contingency plans). 



Smorthing the way has already been discussed 
briefly in the context of involving teachers and ad- 
ministrators in evaluation planning and implemen- 
tation. Involvement and communication are the 
two most important keys to smooth operations. 
You should: 



Anticipating potential 
problems. 



9 



solicit evaluation questions from all persons 
who have a stake in the project (including 
parents). 

disseminate information about the project and 
how evaluation can lead to its improvement, 
inform all concerned parties well in advance of 
scheduled evaluation activities that may require 
their time or disrupt their schedules, 
provide teachers and administrate s with 
prompt feedback regarding test scores, 
train all data collectors to be as nonintrusive as 
possible. 

perform quality-control checks on all data col- 
lectors. 

be responsive to all questions and criticisms, 
report all findings in a timely manner. 



The remaining chapters of this Users' Guide include 
descriptions of many of the kinds of problems you 
may encounter in implementing your evaluation. 
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You should read these discussions carefully as 
problems often do arise unexpectedly, and you need 
to recognize them quickly and respond ap- 
propriately. Inappropriate or delayed responses 
could threaten the validity of an entire yearns 
evaluation. 

This Users' Guide has been developed in the belief 
that the evaluation of bilingual education projects 
can accomplish much more than a minimal satisfac- 
tion of legislative requirements. Thoughtful plan- 
ning and careful attention to evaluation practices 
are necessary, however, if evaluation results are to 
be useful in improving the education of language 
minority students. The remaining chapters of the 
Guide will help the evaluation team plan ap- 
propriate evaluation activities. 
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IV. COLLECTING PROCESS DATA 

Purposes of 

process data. Section 500.51 of the current evaluation regulatioas 

requires that you collect data on student, project, 
and staff characteristics. We recommend that you 
also collect data on any events that may affect 
project outcomes. These data will serve two 
purposes: 

• documenting discrepancies between what was 
intended (process objectives) and what actually 
occurred. 

• documenting project and context characteristics 
for scientific and accountability purposes. 

The necessity of collecting data for scientific and 
accountability purposes is easily understood. In- 
formation related to process objectives, however, 
should be directly useful to you in working toward 
project improvement. 

Research has demonstrated that a new project is 
rarely fiiUy implemented in its first years. Some 
teachers may be following the project's curriculum 
quite closely, but others are likely to be deviating 
substantially from intended practices. Still other 
teachers might be following project plans as far as 
using LI to teach math, but might be inadequately 
proficient to use LI exclusively in teaching science. 



Full implementation is 
unlikely in first years. 
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Deviations from the original project design can be 
adaptive or maladaptive. In either case, outcome 
data will be easier to interpret if you know, for ex- 
ample, that science was taught partially in English 
rather than exclusively in Vietnamese or that LI 
language arts textbooks did not arrive until January. 
Outcome data cannot be used for program im- 
provement unless actual project circumstances and 
events are known. 

Steps in collecting 

process data. There are four steps to the collection of process 

data: 

0 specifying the data to be collected and 
reported. 

o selecting sources and methods for collecting the 
data. 

• preparing to collect the data, 
a collecting and recording the data. 

Each of these steps is explained in more detail 
below. 

Specifying the Data to be Collected 

You will be collecting data to meet the require- 
ments of the regulations and for internal use in your 
project. While there will be some overlap between 
these two types of information, we will discuss thciDi 
separately. 
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Student characteristics data* The current regula- 
tions require that you collect data on the educa- 
tional background, needs, and competencies of the 
limited English proficient persons served by the 
project. [§ 500.51(a)] 

The regulations do not specify exactly wliat data 
must be collected, but we consider the following list 
to be minimal: 

• age. 

• grade level. 

• first language. 

• ethnicity. 

0 language used in the home. 

• proficiency in LI (including literacy). 

• proficien<y in English. 

• socioeconomic status (e.g., participation in Na- 
tional School Lunch Program). 

Ideally, these data should be collected for each stu- 
dent and reported using sunmiarv statistics. 

If necessary, describe 

community characteristics. If certain data (e.g., socioeconomic status) are too 

difficult to collect for individual students, and you 
are serving a homogeneous group, a description of 
the community might suffice. For example, 
"students served by our program come from a 
community where over half the families receive Aid 
• to F^^milies with Dependent Children. Most of the 
students* parents have had fewer than five years of 
schooling and are not literate." 
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Required data on student 
characteristics. 



Additional data that 
may be useful 



There are other data that would be useful to you in 
planning your project or modifying it to increase its 
effectiveness. Such information is also useful to 
audiences of your report because it helps to explain 
why your project is designed as it is. These data 
might include: 



• prior education, including years of schooling 
and possibly type (e.g., monolingual English 
classroom). 

• parents' literacy in LL 

e parents* proficiency in English* 

You should strive to collect all student characteris- 
tic information that will be useful to readers of your 
report in understanding your project. If project stu- 
dents have characteristics that are central to the 
project design, but are not listed here (such as 
giftedness) these characteristics should certainly be 
included even though they exceed minimal report- 
ing requirements. 



Required data on 
length of services. 



Project characteristics data. The current regula- 
tions require that you collect data on the amount of 
time (in years or school months...) participants 
received instructional services in the project and, as 
appropriate, in another instructional setting. 
[§ 500.50(b)(3)(ii)(A)] 



The minimal data needed to satisfy this require- 
ment are: 
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© date of entry into the project. 

• date of exit from the project. 

0 (if project participation is for less than full time 
or less than a school year) other instructional 
settings in which the student is served, such as 
mainstream classroom, special education. 
Chapter 1, etc. 

These data should be collected for each student and 
reported using summary statistics. 

Required data on 

educational activities. The regulations also require that you collect data 

on: 

e The specific educational activities undertaken 
pursuant to the project. [§ 500.51(b)] 

• The pedagogical materials, methods, and tech- 
niques utilized in the program. [§ 500.51(c)] 

0 With respect to classroom activities, the relative 
amount of instructional time spent with students 
on specific tasks. [§ 500.51(d)] 

The minimal data needed to satisfy these require- 
ments are: 

% subject areas included in project instruction. 

• major curriculum objectives. 

• a listing of project materials different from 
those used in mainstream classrooms. 

e percentage of instructional time devoted to LI 

language arts. 
0 instmctional content areas taught in LI. 
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e length of time students are expected to remain 

in the project. 
© for each major curriculum objective, the hours 

per year at each grade level devoted to the 

objective. 

To illustrate the final item above, you might report 
for an objective such as Accurate addition of single- 
digit numbers, "0 hours in kindergarten, 36 hours in 
first grade, 40 hours in second grade, 20 hours in 
third grade." 

Additional data on 

educational activities. Additional data may be necessary to describe your 

project accurately, such as the use of aides, peer 
tutoring, parent education, or computer-assisted 
instruction. 



Staff characteristics data. The current regulations 
require that you collect data on the educational and 
professional qualifications, including language com- 
petencies, of the staff responsible for planning and 
operating the project. [§ 500.51(e)] 

Required data on 

staff characteristics. We consider the following items to be the minimum 

that will satisfy the requirements: 
a level of education, 
e credentials and certificates. 

• bilingual/bicultural teaching experience. 

• other teaching experience. 
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Collect data to check on 
process objectives. 



o languages understood and spoken, including 
degree of fluency, and whether English is 
spoken with a heavy accent. 

Data on project implementation* In addition to the 
process data you must collect to satisfy the regula- 
tions, you will also want to collect data that will tell 
you whether your project is operating as originally 
intended. If you find that certain activities are not 
taking place as planned, you vAW have to determine 
whether the changes represent an improvement or 
whether the project would work better as it was 
originally designed. 

Your project's process objectives should teil you 
what, if any, additional process data should be col- 
lected for use in your evaluation. Ideally, you 
would collect data to verify that each objective had 
been met. Realistically, you may have to review 
your objectives for critical project features or ac- 
tivities that are not covered in the minimal report- 
ing requirements listed in the previous section. 
Thus, if a process objective specified that students 
will be grouped in classes according to their LI 
literacy, you would want to collect information on 
student grouping. Then an examination of students' 
literacy in LI (data that must be collected anyway 
to satisfy the regulations) will tell you if students 
were grouped as planned. 
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Areas of possible 

data collection. Each project will have its ov^n process objectives 

and critical features. It is not possible to specify 
here what these features will be. However, some 
areas to keep in mind while thinking about critical 
activities or objectives are: 



• parent involvement. 
9 Staff development 

» materials development. 

• patterns of classroom language use. 

• use of aides, resource teachers, or other staff. 
% use of special equipment or materials. 

• integration of students' home cultures into 
classroom activities. 

e use of specific classroom management 

techniques. 
^ methods of project management or 

coordination. 



Questions to determine 

important project features. As an additional help in determining what project 

features you will want to document, you can address 
these questions: 



0 In what ways do you expect a project classroom 
to differ from a mainstream classroom at each 
grade level? 

• Wliat special or additional services does your 
project provide that are not normally provided 
by the school district? 



® What aspects of the project are intended to 
meet needs that are unique to the target 
population? 



Plan ahead for data 
interpretation. 



In order to interpret student outcome data, you will 
need to know the degree to which the project has 
been implemented as intended in critical areas. 
Therefore, you must plan to document critical 
processes even if the documentation goes beyond 
that required by the regulations. 



Selecting Data Collection Methods 



Consider three data 
collection methods. 



Three general data collection methods are dis- 
-lussed in this section. They are: 



• examination of records. 

9 self-reports by project staff. 

• observation of project activities. 



If possible, use 
existing records. 



You will probably want to use some combination of 
these methods to collect process data. 



Examining records. Once you have determined 
what data you want to collect, you should inves- 
tigate the possibility that the data have already 
been recorded somewhe^^ in district, school, or 
classroom files. Characteristics of students and 
qualifications of staff are two reporting areas that 
probably have extensive documentation you can 
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use. Some project activities will also be docu- 
mented in classroom records, although in most 
cases you will find that record-keeping is not 
uniform across teachers. 

Potentially useful records. Records which can provide useful process informa- 
tion are: 



9 

% 

o 



6 
0 



Student files. 

teachers' daily activity logs, 
lesson plans. 

records of homework assignments. 

records of special assistance provided to certain 

students. 

minutes or agendas from parent advisory coun- 
cil meetings. 

in-service training announcements or schedules, 
project personnel resumes and/or applications. 



It may also be possible for teachers to aeate or 
augment certain records in ways that will be useful 
for evaluating project processes. They could, for in- 
stance, be asked to include language of instruction. 



Self-reports can be 
questionnaires^ interviews, 
or written reports. 



Self-report measures. You can ask project stafif for 
information about activities not recorded in any ex- 
isting document. Self-reports can be reliable 
sources of data, but they are likely to be biased if 
the questions are value-laden (e.g.. Did you do what 
you were supposed to do?). Self-reports, as long as 
they are not self-evaluations, can be very useful 
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sources of information-particularly if they can be 
supported by other data (such as classroom 
observations). 



There are three methods of collecting self-reports. 



a Staff can write periodic reports of what they did 
(e.g., a monthly report of the percent of time 
lessons were taught in LI). 

9 Questionnaires can be given to staff to fill out 
(e.g., a questionnaire every semester about 
courses or workshops attended). 

o Staff can be interviewed in person (e.g., a year- 
end interview investigating how information 
about students' home cultures was incorporated 
into classroom instruction). 



There is no substitute for 

classroom observation. Classroom observation. Formal or informal class- 
room observation should be done to verify self- 
reports and to document classroom processes. In 
order to determine whether the project is being 
implemented as planned, there is no substitute for a 
visit to the classroom. At a minimum^ you should 
informally visit classrooms during times when 
specific subjects are being taught. 

Plan for even informal 

observation. Before each visit, you should ask yourself what you 

expect to see. Should students be using locally- 
developed materials? Should the teacher be using 
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Design a form for each 
record or purpose. 



English v/ith a carefully selected vocabulary? 
Should a team teacher be previewing instruction in 
LI? Should an aide be translating instruction for 
students who are having difficulty? 

If you see exactly what you expected to see, there is 
reason to believe the project is being implemented 
as intended. If not, you should find out why. Pos- 
sibly the project as designed was not working for 
the students, the teacher, or both. Possibly, the 
teacher needs more training or more time to be- 
come comfortable with the project. To assist in 
determining why discrepancies are observed, some 
form of self-report may be useful. 

Developing Instruments for Collecting and 
Recording Data 

For even the most minimal data collection effort, 
you will need to develop forms to record and store 
your data. You will need a form for each student 
and for each instructional staff member in order to 
record data required by the regulations. The forms 
should be of your own design, depending on the 
data you decide to collect. Make sure, however, 
that forms are clearly labeled and dated and that 
each student record includes the students full name 
and district identification number. 

Forms for record extraction. If you vAW be extract- 
ing process information from records such as agen- 
das of parent advisory council meetings or teachers* 
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lesson plans, you will need to design a form for each 
type of record. Make sure that each form is clearly 
labeled and dated, and if someone else will be fill- 
ing in the form, that it includes complete 
instructions. 

Forms for self-report data* To develop an instru- 
ment to collect self-report data, you will need to 
decide: 

• the type of self-report you want (written report, 
questionnaire, or interview). 

9 whether you want to collect data periodically or 
only once per year. 

• whether you will have time and money to 
analyze individual written or oral responses or 
whether you want to construct closed-choice 
questionnaires. 

Brief questionnaires can 

be easy and economical Table 2 contrasts the types of questions suitable for 

each type of report. For economy of data collec- 
tion, processing, anu aijalysis, a brief, closed-choice 
questionnaire is the best choice. This type of ques- 
tionnaire requires careful construction, however, to 
make sure that the response alternatives you offer 
reflect everything your respondents would like to 
say. The first time you use such a questionnaire, it 
is best to allow respondents to write in answers if 
they wish. This will enable you to see if your 
choices need to be revised. 
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Table 2 

Examples of Self-Report Items 



Type of Instrument 



Self-Report 



1. Describe how you 
used your classroom aide 
this term during English 
language arts instmctioa 



Questionnaire 



Row did you use your 
classroom aide this term 
during English language 
arts instruction? Rank 
the three most frequent 
activities (1 = most 
frequent). 

correct paperwork 

translate for class 

work with small 

groups 
clerical and admin- 
istrative work 
instruct class 
handle discipline 
problems 



Interview 



How did you use your 
classroom aide this term 
during Eng^sh language 
arts instruction? 

Probes: Did you use the 
aide to help with 

^? Why or why 

not? 



2. Compare your use of 
LI and English this year 
for instruction in science. 



At the beginning of the 
year, what language(s) 
did you use for instruc- 
tion in science? 

mostly LI 
half LI and half 
English 
^mostly English 

At the end of the year, 
what languages did you 
use for instruction in 
science? 

mostly LI 
half LI and half 
English 
^mostly English 



How did your use of 
English and LI for in- 
struction in science 
change over the year? 
Why? 
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Types of observation 



instruments. 



Forms for classroom observation. For even infor- 
mal classroom observations, you will need a simple 
form on which to record your observations. If ob- 
servations will be more formal or will be done by 
more than one person, you will need a more 
detailed instrument. There are three common 
types of observation instruments: 



• behavioral checklists (on which the observer 
tallies the number of times he or she sees a 
specific behavior). 

• coded behavior records (on which sequences of 
specific behaviors can be coded). 

• delayed report instruments (which obsen^ers 
use to describe what they have seen over a 
short period of time). 

Each of these observation methods is discussed 
briefly in Appendix A, along with its advanta^^es 
and disadvantages. You should give each careful 
consideration before deciding which is likely to be 
most appropriate for your project. 



Focus on key project 
features. 



Regardless of the particular method of classroom 
observation that is employed, it is simply not pos- 
sible for a single observer to see everything that 
goes on in a classroom. For this reason, the best 
strategy is to focus observations on those actois and 
activities that are most critical to the intended im- 
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plementation of the project and to its success. 



If the project designer has specified that all instruc- 
tion should be conducted in English and has 
developed outcome expectations on the presump- 
tion that it will be, clearly the language of instruc- 
tion must be a focus of classroom observations. 
Other examples might include strict adherence to a 
hierarchy of instructional objectives, or immediate 
positive reinforcement of all correct student 
responses. It should be noted, however, that fea- 
tures considered critical to the proper implementa- 
tion of one instructional design (and therefore to its 
success) may be of little concern to another. 

If a project's design has been well and fully expli- 
cated, it should be possible to identify the most 
critical things to observe. Even when this is the 
case, however, it would be good practice to review 
classroom observation plans with the project 
designer. With a less well explicated project design, 
such review is absolutely indispensable. 



server to assess a classroom's "climate," you are 
going to get a highly subjective response and one 
that would not be consistent across different ob- 
servers. One observer might describe it as "noisy, 
chaotic, and out-of-control", while another might 
see it as "indiv'dualized, spontaneous, and creative." 
To avoid this kind of subjective inconsistency you 



Avoid high-inference 
observations. 



If, after some period of observation, you ask the ob- 
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should make the observational task as objective as 
possible. 



You will get better results by having your observer 
count the number of positive reinforcements r.ade 
by the teacher in a 20-minute period than by having 
him or her rate the teacher on a 7-point scale from 
positively to negatively reinforcing. Even more 
reliable data will be obtained if the observer does 
no more than tally each time the teacher uses the 
word "good," since simply deciding what constitutes 
an instance of positive reinforcement requires a 
moderately high level of inference. 



Multiple observers must 
agree on what they see. 



If all classroom observations are done by a single 
observer, high inference observations are less 
problematic. When multiple observers are used, 
however, it is essential that they be trained to the 
point that they have a high level of agreen^ iit 
(interrater reliability) regarding what theyst^. 
Adequate interrater reliability (see Appendix B) 
will be much easier to obtain with low-inference ob- 
servation tasks* No amount of training n:.ay 
produce adequate interrater reliability if the obser- 
vation tasks require high levels of inference. 



Preparing for Data Collection 



Make sure data collectors 
know what to do. 



Different data collection me.thods will require 
slightly different preparatory steps. In genera^, you 
will always need to: 
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« select and train data collectors (unless you plan 

to collect all the data yourself), 
© check data collectors' early efforts to make sure 

data collection is done correctly. 

For classroom observation, these activities are more 
complicated than for record extraction. For train- 
ing observers, you m\\ need a videotape of a class 
or a live nonproject class which can be observed for 
practice. You mil also need to verify interrater 
reliability if more than one observer is used. (See 
Appendix B.) 

Collecting and Recording Process Data 

Once you have developed forms to collect all the 
data you need and you have trained your data col- 
lectors, data collection should be straightforward in 
most cases. For some projects, most data collection 
wii^ be accomplished toward the end of the school 
year. In other instances, data will be collected 
throughout the year, e.g., as new students enter the 
project. If data are collected as events occur, they 
should be checked and consolidated every few 
months. 

In collecting classroom observation, data, there are 
additional poims to remember: 

o observe each classroom several times during 
the year. 
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9 look for different features in different observa- 
tion sessions-don't try to accomplish too much 
in one session. 

• avoid scheduling observations at times when 
classroom processes will be atypical, such as 
near a holiday. 

c make sure observers remain unobtrusive and 
do not disturb classroom activities. 

Other events can affect 

project outcomes. Recording non-project data. During the course of 

the school year, events may occur which affect 
project outcomes but which are unrelated to project 
objectives. Such events might include a teachers' 
strike, lengthy school closings because of bad 
weather, or a locally upsetting event such as a fac- 
tory closing. By keeping in touch with project staff, 
you can j5nd out about external events that are 
likely to affect student performance. You should 
keep a record of all such events, noting the facts of 
the events, the dates, and possible impact on stu- 
dents. You can refer to this written record when 
you interpret data. 
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Y. SELECTING/ADAPTING/DEVELOPING INSTRUMENTS FOR 
ASSESSING OUTCOME OBJECTIVES 



Poor test choices make 
projects look ineffective. 



It is essential to pay careful attention to the task of 
choosing appropriate tests for evaluating bilingual 
projects. As we point out below, it is possible to 
make several different lands of poor choices. Un- 
fortunately, most poor choices will have the effect 
of making interventions appear to be less effective 
than they really are. Perhaps even more unfor- 
tunate is the fact that good test choices may be 
precluded by district or state rules and regulations. 



With respect to the latter point, local-level evalua- 
tions must conform to the district and state regula- 
tions. If the required practices are unsound, 
however, and especially if they make projects ap- 
pear less effective than they really are, tactfiil ef- 
forts to bring about district- or state-level change 
may have long-range benefits for all concerned 
parties. 



Content validity is the 
mcrt important selection 
ciitenon. 



High reliability and validity arc always considered 
to be desirable test characteristics, and they are 
specifically required by Section 500.50(b)(2)(ii) of 
the current evaluation regulations. Appropriate dif- 
ficulty levels and freedom from cultural bias are 
also considered to be meritorious features. For our 
purposes, however, a particular form of validity- 
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content validity-wUl be of primary concern. As we 
shall see, a test that has high content validity will 
tend to have the other desirable characteristics as 
well. 

Content Validity 

Content validity means 

measuring the right stuff. The content vaUdity of a test is the extent to which 

that test taps the skills and knowledge it is supposed 
to measure. Content validity is always assessed by 
means of subjective impressions of item relevance. 

In general, it can be said that a pretest has content 
validity if it accurately reflects the performance 
level of students who are about to enter a new in- 
structional sequence. To do this, the test must 
measure both what students have akeady been 
taught and what they will be taught in the new 
sequence. It must measure both of these areas be- 
cause some students will not have learned every- 
thmg they were taught previously and others will al- 
ready know some of what vdll be taught in the new 
sequence. 

Aposttest can be thought of as the pretest for the 
next instructional sequence. Thus, to have content 
validity, a posttest must measure what was taught in 
the previous instructional sequence as well as what 
will be taught in the next instructional sequence. 
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Content validity and 
difficulty are closely 
related 



At this juncture it is appropriate to point out that 
there is a close relationship between content 
validity and difficulty. If a test has high content 
validity, it should be neither too difficult nor too 
easy. 



Since too easy and too difficult tend to have subjec- 
tive meanings, we offer the following operational 
definitions: 



Definitions of too 
difficult and too easy. 



A test is too easy if any of the testees know the 
answers to all of the questions. When this is 
the case, one can assume that some students 
would know the answers to additional, even 
more difficult questions. 



e A test is too difficult if any of the testees do not 
know the answers to at least some of the ques- 
tions. When this is the case, one can assume 
that some students would not know the answers 
to additional, even easier questions. 



It is important to note that the preceding definitions 
use the phrase "know the answers" rather than 
"answer the questions correctly." Items may be 
answered correctly by random guessing, or 
answered incorrectly because of careless errors. 
Thus, tests can be too difficult even if there are no 
zero icores or too easy even if there are no perfect 
scores. 
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Assess the match of test 
items to curriculum 
content 



Assess difficulty to verify 
content validity. 



Project evaluation imposes 
four content-validity 
requirements. 



Our earlier statement-if a test has high content 
validity, it should be neither too difficult nor too 
easy-should not be taken to imply that tests which 
are neither too easy nor too difficult have high con- 
tent validity. Content validity can only be assessed 
by comparing the test questions with the cur- 
riculum. If there is a good miatch, then there is high 
content validity. 



Looking at difficulty levels is a useful check on con- 
tent validity-and if tests are found to be too easy or 
too difficult, something needs to be done to correct 
that problem-but checking difficulty levels is no 
substitute for a direct assessment of content 
validity- 



Before discussing procedures for assessing content 
validity, it is appropriate to examine in some detail 
the specific content-validity requirements you may 
have to address in evaluating bilingual education 
projects. Because the regulations specify that the 
progress of project participants [must be] measured 
against an appropriate nonproject comparison group 
[§ 500.50(b)(1)] evaluation instruments need to 
have content validity for four different data points: 



• the pretest performance level of the project 
group; 
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© the pretest performance level of the com- 
parison group; 

• the posttest performance level of the project 
group; 

© the posttest performance level of the com- 
parison group. 



Because comparison-group students may be sub- 
stantially higher achievers than project students, 
these content-validity requirements may present 
difficult instrumentation problems— at least for 
some academic subjects. 



Range of test coverage is 
a potential problem. 



Even at the elementary grades, it may be impossible 
to jSnd a single test that covers the range of perfor- 
mance levels from that of the project group at 
pretest time to that of the comparison group at 
posttest time. At the later grades, the problem will 
be much more severe. Especially in the area of 
English language proficiency, tests that are ap- 
propriate for one group may be totally inap- 
propriate for the other-both in terms of content 
and in terms of difficulty. 



Functional level testing 
is a potential solution. 



There is only one adequate solution to this 
problem-testing the two groups with different levels 
of the same test hatter^. This strategy is called func- 
tional level testing, and it is a psychometrically 
sound method for measuring all four performance 
levels (see above) on a common scale. Appendix C 
presents a brief discussion of functional level testing 
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English language 
proficiency is broadly 

defined English language proficiency. English language 

proficiency is a term that the regulations and the 
Users^ Guide use in the broadest possible sense to 
encompass understanding and speaking English, 
reading readiness, reading, and possibly even writ- 
ing. It is thus not necessary that you employ a test 
labeled "language proficiency" for your evaluation. 
In fact, readiness and reading tests will usually serve 
evaluation purposes more effectively. 

Language proficiency tests tend to have as their 
primary objective the classification of students as 
either LEP (needing bilingual services) or non-LEP 
(able to function adequately in mainstream 
classrooms). They are generally not well designed 
for project evaluation purposes and we recommend 
that you use them only when other instruments are 
clearly inappropriate. 



Native language proficiency. Native language 
proficiency need be assessed only in projects of 
developmental bilingual education, although such 
assessment may be useful for other types of projects 
as well. 

Suitable tests may be 

hard to find The main difficulty with assessing native language 

proficiency is the dearth of suitable instruments- 
especially for languages other than Spanish. 
Several English language reading and reading 
readiness tests have, however, been professionally 
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translated into Spanish, Vietnamese, and other lan- 
guages. Your regional (East or West) Evaluation 
Assistance Center should be able to help you iden- 
tify these instruments. 

For less commonly encountered languages, you may 
experience substantial difficulties in locating 
suitable tests. Only locally developed instruments 
may exist and they may be difficult to find. Again, 
however, one of the Evaluation Assistance Centers 
should be able to help. 

If no suitable off-the-shelf tests can be found, you 
will have to develop your own or translate an 
English language test. Unfortunately, neither of 
these options is particularly desirable. 

Translating tests is 

difficult and hazardous. Translating tests is a very difficult task and one that 

is likely to be successful only with instruments 
covering very basic skill levels. Picture-word as- 
sociations may qualify, but paragraph meaning 
questions will almost certainly present serious 
problems to even the most expert translators. 

Translating from English into a non-Indo-European 
language will be particularly hazardous. You would 
probably be better off developing a test from 
scratch if you are working with Asian or Native 
American languages. Even with an Indo-European 
language, your chances of producing a successful 
translation will depend on the extent to which the 
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English version of the test has the following 
characteristics: 



e short questions. 

• active voice. 

• specific rather than general terms. 

# no metaphors or colloquialisms. 

# no vague words (probably, frequently, 
sometimes). 

e no subjunctive mood. 



Seek professional help in 
developing new tests. 



Compare progress in LI 
to progress in English 



If the test you wish to translate has these charac- 
teristics. Appendix D provides some guidance on 
how to proceed. 



As far as developing new tests is concerned, we 
recommend that you seek the help (not just the 
advice) of a professional expert in psychometrics. 
You should not consider using any instrument un- 
less it has been item-analyzed and revised on the 
basis of that item analysis. This recommendation 
pertains to tests developed elsewhere as well as to 
any you may develop locally. 



Assessing progress in native language proficiency 
relative to a nonproject comparison group appears to 
mean comparing the project group's progress in LI 
with the comparison group's progress in English. 
This type of comparison requires either a test writ- 
ten in LI that has national norms or a test without 
norms that has content validity for both the project 
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group's LI curriculum and the comparison group's 
English curriculum. This issue is discussed more 
fully under the following heading. 



Suitable tests may again 
be hard to find 



Other academic subject areas. Achievement tests 
for assessing progress in non-language subject areas 
should be written in the language of instruction un- 
less the students are more proficient in English. 
Again, there may be problems in finding tests that 
have content validity for the particular curriculum 
being taught and are written in the language of in- 
struction. Your regional Evaluation Assistance 
Center is probably your best source of help* 



Using norms with LI 
translations of 
standardized tests. 



An important consideration in selecting subject 
matter tests is how you will use them in your 
evaluation. You will need to quantify the pre- and 
posttest performance levels of both the project 
group and a comparison group. If you can find a 
suitable test that has national norms, you may use 
the 50th percentile of these norms as your com- 
parison group. 



Special equatings of the LI and English versions of 
the test may have been done by the publisher, and 
special provisions may have been made for access- 
ing the norms. Special norms may even have been 
developed for the LI version of the test. Even if 
nothing special has been done, however, the gap- 
reduction design (see Chapter VII) wih work quite 
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well using the raw scores obtained by project stu- 
dents on the LI version of the test and the raw 
scores corresponding to the 50th percentile of the- 
appropriate English language norms- Whatever 
biases may exist in the pre- and posttest 
performance-level indicators cancel out when rela- 
tive growth is what is being assessed. 

Using live comparison 
groups requires curriculum 

comparability. if you cannot find a suitable test with national 

norms, you will need to use an unnormed test with 
the mainstream grade mates of project students as 
your comparison group. Two conditions must be 
met, however, whenever mainstream grade mates 
are used as the comparison group: 



o there must be no significant differences be- 
tween the curricula being taught to the two 
groups, 

o there must be both English and LI versions of a 
test that has content validity for the common 
curriculum. 



• If the two curricula are significantly different, 
you have only two choices. You may use an 
English language normed test (untranslated) or 
translate such a test into LI. 
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If the project students have some proficiency in 
English (even though they are taught in LI), it may 
be possible to test them using an instrument written 
in English. Under these circumstances, it will prob- 
ably be appropriate to make some modifications to 
the test so that it more accurately reflects the 
student's subject matter knowledge. 

Three potential problems. When tests are used with groups other than those 

for which they were designed, three kinds of 
problems are likely to be encountered: 

• students will fail to understand what they are 
supposed to do. 

• students will not have time to respond to ali the 
items because of their slow reading. 

• students will fail to understand some questions 
because of unfamiliar English words that are 
totally unrelated to the knowledge being 
tapped. 

Three partial solutions. All three of these problems are at least partially 

fixable. You can: 

:^ Modify instructions to be sure they are clear to 
your project students. You may translate then 
into LI or paraphrase them in LI. 



Using English language 
tests for subjects taught 
in LL 
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Modifying tests may 
enhance reliability and 
validity. 



d Extend time limits so that most students will 
have time to attempt most of the items on the 
test. This is just what the test developer had in 
mind when the time limits were established for 
non-LEP, mainstream students* 

€ Simplify the language of the test items-but only 
to the extent that the words or phrases 
simplified are independent of the content being 
tested. If you want to test the students' math 
skills, don't make them ':ake a reading test in 
order to get to the math items. 



Making the kinds of changes just discussed will ac- 
tually enhance the reliability and validity of the test 
for your project students. It will only do so for raw 
scores, howeven If the test has norms, they will not 
be valid for students taking the modified version of 
the test (although you can still use those norms as 
your comparison group in the gap-reduction 
design). 

Modifying tests will also invalidate any interlevel 
equating. You should not use modified out-of-level 
tests with either of the recommended comparison 
groups (norms or mainstream grade mates). 

Assessing Content Validity 

As mentioned earlier, the content validity of a test 
is always assessed in terms of the extent to which its 
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items tap the skills and knowledge the test is sup- 
posed to measure. It is not possible to make such 
an assessment from information provided by the 
test publisher or test information service centers* It 
is absolutely essential for persons familiar with your 
curriculum to examine candidate instruments on an 
item-by-item basis. 

Identifying candidate 

instruments. Before you begin the content validity assessment 

process, you must, of course, identify a group of 
candidate instruments. We recommend that you 
begin by contacting the major publishers of stand- 
ardized achievement tests (see Appendix E) and 
obtain "specimen sets" of instruments that each 
publisher's sales representative feels would be ap- 
propriate for your target group. You should also 
contact your regional Evaluation Assistance Center 
for additional nominations. Finally, if it has not al- 
ready been identified as a candidate, you should 
consider any instrument that is administered to your 
students for statewide or districtwide testing pur- 
poses. 

Recommended procedures 
for assessing content 

validity. W^ile this task may appear quite burdensome, it 

should not be so in prrxtice. In fact, il should be 
possible for a person who is familiar with the cur- 
riculum to perform a content analysis in ap- 
proximately the same amount of time that person 
would require to take the test The procedure we 
recommend is for the test evaluator to read each 




question and to mark on a regular IBM-type answer 
sheet whether: 

• the item covers material that was taught to 
project and comparison group studeats prior to 
the project year that is just beginning (you may 
use answer alternative "a" to mdicate such 
items)* 

0 the item covers material that will be tauglit 
during the project year that is just beginning or 
about to begin (use answer alternative "b" to 
indicate such items). 

€ the item covers material that will be taught in 
years subsequent to the project year that is just 
beginning (use answer alternative "c" to indi- 
cate such items). 

• the item covers material that is not included in 
the past, present, or future curricula (use 
answer alternative "d" to indicate such items). 

Ideal distributions 

of item types. Ideally, at pretest time, you would like to find that 

about 30% of the items were type "a", about 40% 
type "b", and 30% type "c". Type "d" items serve no 
useful purpose in your evaluation. Consequently, 
you would like to find very few or none of them. 

At posttest time, the ideal distribution would be 
approximately 70% of the items being classified as 
types "a" or "b" and the remaining 30% as type "c". 
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Problems will result from 

missing item types. Clearly, these ideal distributions will not be exactly 

matched by any existing tests. On the other hand, 
tests must have at least some items in categories "a", 
"b", and "c'' in order to be well suited for evaluation 
purposes. 

• Tests that lack type "a" items will be too dif- 
ficult at pretest time and will yield 
performance-level estimates that are systemati- 
cally too high. 

• Tests that lack type "b" items will be insensitive 
to project-related learning. 

• Tests that lack type V items will be too easy at 
posttest time and will yield performance-level 
estimates that are systematically too low. 



Reliability 



Test reliability is an extremely important considera- 
tion when assessing individual students. When as- 
sessing the progress of groups of students it be- 
comes much less critical. Still, with typical, project- 
size groups, more reliable tests should be preferred 
over less reliable tests, all other things being equal. 

Three relevant reliability 

considerations. There are several reliability facts that you should 

consider: 
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e The reliability of a test is proportional to its 
length-but the length of a test is defined by the 
number of items that students respond to, not 
the number of items printed on its pages, A 
functional level test will be more reliable (and, 
as we have already discussed, more valid) than 
a test made up of items students can only guess 
at. 

© Standardized achievement tests put out by the 
major test publishers are all about as reliable as 
such tests can be. Locally developed tests will 
usually be significantly less reliable. 

a Test reliabilities change with the characteristics 
of the group tested. They will be significantly 
lower for the kind of homogeneous groups typi- 
cally served by Title Vn projects than for the 
much more heterogeneous norm groups for 
which published reliability coefficients were 
published. 

Reliability is less important 

than content validity. Appendix F provides some guidelines you can use 

to determine the effect of reliability differences on 
the accuracy of growth and gap-reduction estimates. 

As that appendix demonstrates, however, reliability 
differences must be quite large or reliability levels 
quite low before the impact on the accuracy of 
growth estimates amounts to much. For the most 
part, reliability differences will be trivial compared 
to differences in content validity. Perhaps even 
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Cultural bias has little 
effect on measures of 
growth. 



more significant is the fact that tests with high con- 
tent validity will tend to be more reliable than tests 
with low content validity. In the final anr ysis, you 
should probably not worry very much about test 
reliability. 

Cultural Bias 



Cultural bias, like reliability, is extremely important 
when assessing the status of individual students. 
When tests are used for evaluating bilingual 
projects, however, cultural bias is much less impor- 
tant. Whatever bias was present at pretest time will 
also be present at posttest time (although probably 
to a somewhat lesser degree). Thus, when pretest 
scores are subtracted from posttest scores, cultural 
bias tends to cancel out. 

The most widely recognized form of cultural bias 
results from items that are more difficult for 
minority than for majority students. In extreme 
cases they may even have different "correct" 
answers. Most publishers of standardized achieve- 
ment tests employ debiasing procedures to detect 
and eliminate such items. 

While debiasing procedures are generally quite ef- 
fective, they are all less than perfect. Debiased 
tests may still yield somewhat biased status 
indicators; the growth estimates they yield should 
be quite acceptable, however. 
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There is a second kind of cultural bias in tests-a 
bias that stems from mainstream students having 
acquired more test-wiseness than minority LEP 
students. This bias can be reduced by teaching test- 
taking skills to LEP students and instilling in them a 
more competitive attitude toward test-taking situa- 
tions. 



Gains in test wiseness 
confound growth 
estimates. 



An increase in test wiseness will cause test scores to 
go up and cultural bias to go down. Note however, 
that an increase in test wiseness between pre- and 
posttesting will be confounded with whatever in- 
crease in subject matter proficiency occurred over 
the same period. 



If project participants are observed to be catching 
up to their mainstream grade mates in terms of 
scores on an English language proficiency test, that 
catching up may be due partially or entirely to gains 
in test wiseness rather than gains in English lan- 
guage proficiency. You should keep this possibility 
in mind when interpreting your evaluation findings. 

Summary of Test-Selection 
Recommendations 



Use standardized 
achievement tests. 



Whenever possible, we recommend that you use 
standardized achievement tests to evaluate bilin- 
gual education projects. While they have serious 
deficiencies when used for other purposes, they 
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have no equal for measuring growth along well 
defined achievement dimensions. They also tend to 
be more reliable than locally developed tests. 

Four additional 

recommendations. Additional recommendations include: 

§ Use only tests that have high content validity 
ioi both the project group and the comparison 
group in your evaluation. 

^ Use functional level testing as appropriate to 
achieve high content validity. 

d Whenever possible, avoid instruments labeled 
language proficiency tests. They were not 
designed for evaluation purposes and are not 
well suited for such usage. 

o Use the mainstream grade mates of project 
students for your comparison group in 
preference to national norms whenever there is 
a good curriculum match and such usage does 
not impose an excessive testing burden. 
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VI. COLLECTING OUTCOME DATA 



The regulations specify 
12-month testing 
intervals. 



The current evaluation regulations require that you 
collect ' oth test and non-test data. The required 
test data are to be collected on a 12-month cycle 
and include: objective measures of the academic 
achievement of [project] participants related to 
English language proficiency, native or second lan- 
guage proficiency (for programs of developmental 
bilingual education), and other subject matter areas 
[§500.50(b)(2)(iv)]. 



Current and former 
project participants 
must be tested 



These data are to be collected from both LEP and 
native English speaking current project p irticipants, 
and from former LEP project participants who were 
exited and are currently in maia^ tream classrooms. 

To prepare for achievement testing you will need to 
plan the testing schedule, train the test ad- 
ministrators, and prepare the students and setting. 
Each of these steps is discussed below. 

Planning the Testing Scheduk 

In scheduling teste, you should take the following 
important considerations into account: 
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Plan to avoid 
excessive testing. 



Spring-to-spring testing 
requires a separate 
baseline test 



• The regulations require a 12-month testing in- 
terval. If you use a standardized achievement 
test, you can avoid double testing by using each 
year's posttest as the following year's pretest, 
but you must select the best points at which to 
change test levels. 



Usually, you will select either a spring-to-spring 
or a fall-to-fall schedule. If you select spring- 
to-spring testing, however, you will need to 
pretest all 'first year participants in the tall to 
establish a pre-project baseline performance 
level. 



Norm-based 
coyiparisons impose 
special requirements. 



Make provisions 
for testing (nearly) all 
project participants. 



« If you plan to use norms as your comparison 
group, the best testing time is within two weeks 
of the date the test was normed. With a 12- 
month testing interval, you may, however, use 
interpolated norms which the publisher will 
usually provide upon request. 



• The regulations i quire that evaluation findings 
be representative of the students served. This 
means that every effort must be made to test all 
students by: 
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(a) scheduling make-up testing sessions near 
the original testing dates to obtain scores 
for students who were absent. 



b) scheduling special testing sessions for stu- 
dents who enter the project after the 
pretest or leave before the posttest so that 
their scores can be included in the evalua- 
tion, (If fewer than 10% of the project stu- 
dents are late enterers or early leavers, you 
may omit this procedure.) 



Training the Test Admii.^strators 



All test administrators and proctors should be 
trained every year no matter how experienced they 
are. The training should emphasize the following 
points: 



Written instructions should be followed exactly, 
whether standardized or unstandardized tests 
are used. If there are no written instructions, 
you should prepare some to guarantee stand- 
ardized test-administration practices. 



Modified tests require 
written administration 
instructions. 



If modifications have been made to standard- 
ized test instructions, they must be written 
down and followed exactly at pretest and post- 
test times. No changes should be made be- 
tween the two test administrations. 
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Watch for indications 

of invalid test scores. • Administrators and proctors should watch for 

behaviors that may indicate invalid test scores. 
These behaviors include: 



(a) talking during the test. 

(b) looking at other students' answers. 

(c) marking answers at random. 

(d) finishing unusually early. 

(e) not paying attention to instructors. 

(f) leaving many answers blank. 

(g) leaving the room. 

(h) losing their place on the answer sheet. 



Proctors or administrators should make a note if 
they observe any of these behaviors during testing. 
Student test scores judged to be invalid should be 
discarded. 



Preparing f he »Students for Testing 

Scores will be more 
valid if students are 

prepared for testing. All students, but LEP students in particular, can 

benefit from thorough preparation for testing. 
Preparation should include: 

• telling students when, where, and why they will 
be tested. 



9 helping students gain test-taking experience by 
allowing them to practice on similar tests and 
answer sheets, including any practice test that 
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T}j to find a clearly 
quiet, well lighted place 
for testing. 



may be provided by the test publisher. 

• teaching students test-taking skills such as 
guessing, saving difficult items until easier ones 
have been answered, budgeting time, etc. 

• encouraging students to do their best on the 
test. 

• notifying parents of the testing schedule so that 
they c'c:a help by making sure students are well 
rested and well nourished on testing days. 

Preparing the Testing Setting 



The testing setting can have a major impact on test 
scores. Idedly, the setting should have all of the 
following characteristics. Since our main concern is 
with change from pre- to posttest, however, the 
most important consideration is that the testing set- 
ting be the same at pre- and posttest times. 

9 The testing space should be clean, quiet, well 
lighted, and large enough to prevent crowding. 

• The same space should be used for every test 
administration; pretesting, posttesting, and 
make-up testing. 
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Standardize testing 
conditions and 
procedures. 



• The space should be isolated from external 
noises such softball games, band practice, 
police sirens, or lawn mowers. 

0 Noises such as a telephone, school bells, or the 
public address system should be silenced during 
testing. 

% Test administrators and proctors should avoid 
noisy shoes or jewelry and should never whisper 
or converse during a testing session. 

€ Desks or tables should be large enough to hold 
both test booklets and answer sheets. 

• Desks or tables should be far enough apart to 
discourage copying and to allow proctors to 
move around the room. 

Administering Achievement Tests 



When tests are administered it is important to 
maintain standard conditions ^or the pretest, post- 
test, and any make-up sessions in order to be sure 
your re.^ults can be interpreted. It is best to follow 
these suggestions: 

• Always go over the practice examples with the 
students before beginning the test. 
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® Adhere strictly to administration directions and 
times. 



e Note any unusual student behavior or any dis- 
ruptions to the testing session. 

Collecting Other Test Data 

Make use of all 

available test data. Although you are not required to by the regula- 

tions, you should collect any other test data that are 
available for your students. These include: 

1 scores from project entry and exit tests. 

6 scores from any state or district testing 

programs, 
o scores from curriculum unit tests, 
e scores from teacher-made tests. 



Collecting Non-Test Data 



Many unobtrusive 
measures are useful for 

evaluation. Non-test indicators can be reliable and easy to ob- 

tain. Several such indicators are required (as 
appropriate) by the regulations. Table 3 describes 
these minimum data requirements. 

We recommend that you also collect other, non- 
required data such as: 



• student mobility rate. 



75 



83 



ERIC 



Tables 

Minimum Requirements for Non-Test Data 



Collect for Each Student Report by Grade Level for Project 

Group 



Years retained in grade as of (date). Average retention rate range, # and % 

of target group with history of reten- 
tion. 



Whether or not student dropped out. 
(Not appropriate for elementary 
grades.) 

Number and percent of days absent 
from project classrooms broken into at 
' least two or three calendar periods. 

Date of special education referral or 
placement. 

Date of placement in gifted and 
I talented program. 

Date of enrollment in post-secondary 
education institution. (Not 
appropriate for elementary grades.) 



#, % of target group (local defmition 
of 'dropout") 

Average percent days absent, range. 

#, % of target group referred or 
placed. 

#, % of target group placed. 
#, % of target group enrolled. 
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• number of disciplinary actions. 
e number of suspensions. 

• project teacher turnover rate. 

© number of books checked out of school 

libraries by project students. 
© numbers of times students were late to class. 

These data can be used as behavioral indicators of 
affective states such as students' self concepts or at- 
titudes toward school We recommend that you use 
such behavioral measures instead of paper-and- 
pencil affective instruments. Such instrumer^s are 
not sufficiently reliable or valid to use in assessing 
project effectiveness. 

Before and after data 

are needed For non-test data, as for test data, you should at- 

tempt to obtain baseline data and then collect addi- 
tional data at least annually. In some cases (such as 
for dropout rates) you may be able to obtain a stat^ 
or national average for your students' ethnic group. 
This average can, if necessary, serve as your 
baseline figure. 

Training Coliectors of Non-Test Data 

Data extractors also 

need to be trained Collectors of these data will abstract them from ex- 

isting records (see Chapter IV for additional guides 
on examining records). They should be provided 
with record abstraction forms and taught: 

• what data to collect. 

© where to find the infoimation. 
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Exercise quality control 
procedures. 



e how to gain access to the information, 
e how and where lo record the data. 
• what precautions to \'ake to protect students' 
privacy. 

Data collectors' work should be checked after they 
have extracted information on a few students. This 
check will show whether additional hands-on train- 
ing is needed. 
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m IMPLEMENTING THE GAP-REDUCTION DESIGN 



Gap reduction is a valid 
criterion for project 
mccess. 



One primary goal of bilingual education is to close 
the gap in English language proficiency between 
project students and their non-LEP peers. A 
sejond primary goal is to keep project students 
from falling behind their non-LEP peers in other 
subjects while they are learning English. 



The gap-reduction 
design is recommended 



It makes sense to talk about bilingual education 
projects and their success in terms cf these gaps. 
Thus it is not entirely coincidental that we have 
developed an evaluation strategy called the gap- 
reduction design which we recomirend for evaluat- 
ing Title VII (and other) bilingual projects. 



Our recommendation is not made lightly. We have 
critically examined all other designs described in 
the literature, the assumptions that underlie them, 
and the implementation requirements tht7 impose. 
While three of those designs (see Appendix G) may 
yield better estimates of project impact if properly 
executed under ideal conditions, none was judged 
to be nearly as easy to implement or interpret as 
the gap-reduction design. This ease of implementa- 
tion and interpretation stems from the design's 
focus on achieving project objectives rather than 
quantifying the size of the treatment effect. 
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The design is 
easy to implement. 



Conceptually, the gap-reduction design is very 
simple. It involves only four, easy-to-measure 
quantities: 



Two comparison groups 
are recommended 



1. the project group's pretest performance 
leveL 

2. the project group's posttest performance level. 

3. the comparison group's pretest performance 
level. 

4. the comparison group's posttest performance 
level. 

From these four quantities, we can easily measure 
the gap between groups at pretest time (#3 minus 
#1) and at posttest time (#4 minus #2). The 
amount of Gap Reduction then is simply the Posttest 
Gap minus the Pretest Gap. Figure 3 provides a 
graphic representation of these relationships. 

Selecting a Comparison Group 

Before going on to discuss the procedures required 
to implement the gap-reduction design, it is ap- 
propriate to consider the issue of selecting a com- 
parison group. There are two possibilities that we 
recommend: 

n the mainstream grade mates of the project 
students. 

• the 50th percentile of the national norms. 
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Figure 3. Illustration of gap reduction. 



Using grade mates as a 

comparison group. Using mainstream grade mates as your comparison 

group has several advantages. 

• You have a local rather than a national basis 
for comparison-a factor that may facilitate in- 
terpretation of the results and make the com- 
parison seem fairer. 

0 You are not restricted to standardized 
achievement tests (although you mcry use 
them). 
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9 You have more freedom in selecting testing 
dates. 

Using mainstream grade mates as your comparison 
group hds only one potential disadvantage-but it 
may be a mjrjcr one. You will have to collect data 
from all members of the comparison group. This 
requirement will not be too burdensome if you can 
"piggyback" your evaluation on an annual dis- 
trictwide or statewide testing program. But if such 
piggybacking means you will be using a test with 
low content validity (see Chapter V), the end result 
will be to make your project appear less effective 
than it really is. 

Using norms as a 

comparison group. When we talk about using national norms, we are 

necessarily confining our attention to achievement 
test'*ng. When the gap-reduction design is imple- 
mented with classroom grades, attendance figures, 
or a variety of other indices, it can only employ 
mainstream grade mates or some other "live" com- 
parison group. Normative data that are statistically 
adequate to use as a substitute for a live com- 
parison group are available solely for standardized 
achievement tests. 

The main advantage of using norms for your com- 
parison grou]: • is that you don*t have to impose an 
additional testing burden on comparison-group stu- 
dents. The only disadvantages-and thf^y may be of 
no concern whatsoever-are that you must use a 
standardized achievement test and you should test 
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within two weeks of the test's empirical norming 
date. 



Other comparison groups 
are possible. 



Although we recommend using either the 
mainstream grade mates of your project students or 
the 50th percentile of the national norms as your 
comparison group, you could use a variety of other 
groups. One example would be a group of unserved 
LEP students from a neighboring school district. 
Many other possibilities exist. 



Gap reduction with a 
variety of measures. 



If you use one or more the these "other" groups, 
your results will be difficult to compare with results 
from other projects. Inteipretation of your results 
will also be more difficult. We urge all Title Vll 
projects to use one of the two recommended com- 
parison groups. Then, if you feel additional com- 
parisons would provide useful information, you are 
encofiraged to investigate and use them. 



The gap-reduct^'on design can be implemented 
using test scores, classroom grades, attendance 
figures, and a variety of other data. Using test 
scores, however, involves some additional computa- 
tional procedures for two important reasons: 



« scores on different tests are non-comparable. 

• test score scales, unlike other scales (e.g., num- 
ber of days absent) do not have equal intervals. 
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Both of these problems must be dealt with mathe- 
matically to provide interpr^table evaluation 
results. 

Assessing Gap Reduction Using Non-Test 

An example using 

attendance data. Before going on to the more complicated issues, it 

will be useful to work through an example of 'gap- 
veduction analysis using attendance data. 

Suppose, that, during their first two months of par- 
ticipation, project students averaged 8.3 days ab- 
sent. During tbe same period, comparison group 
students were absent 3.8 d?*ys. A year later, during 
the same two-month interval, project students were 
absent 5*2 days and comparison group students 3»9» 



Pre 



Post 



Project Group 

Comparison 
Group 



8.3 


5.2 


3.8 


3.9 



Ih^ pretest gap was 4.5 days (8.3 - 3.8). 
The posttestgap was 1.3 days (5.2 - 3.9). 
DxQ gap-reduction was 3.2 days (4.5 - 1.3). 
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Amount vs. percent of 

gap reduction. In this example, there is no real need to "massage" 

the data any further. It would be helpful, however, 
to express the amount of gap reduction as a per- 
centage of the pretest gap. In this case 3.2 days is 
slightly more than 70 percent of 4.5 days. One 
could thus say that the project was successful in 
reducing the attendance-rate gap by approximately 
70 percent after one yean 

The amo 'nt of gap reduction and the percent of gap 
reduction may give quite different impressions as to 
how successful the project has been. It will, there- 
fore, always be a good idea to present both in your 
evaluation report. 

Examining other gaps. Similar analyses could be carried out using class- 
room or report-card grades. This type of analysis is 
particularly appropriate for former LEP students 
who have graduated into mainstream classrooms. 
Whatever gap in grades may exist between the 
former LEPs and their non-LEP classmates shortly 
after mainstreaming should not increase with the 
passage of time. 

If the gap remains constant, the former LEPs are at 
least holding their own. If it decreases, the former 
LEPs are continuing to catch up. But if it increases, 
the project failed to prepare its target students to 
progress effectively through the regular school 
program. 
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Assessing Gap Reduction Using Test Scores 



Test scores may have 

unequal intervals. Test scores, as mentioned earlier, lack some of the 

desirable features that other indices possess. A day 
absent is always a day absent-it makes no dif- 
ference whether it is the second or the forty-second 
day absent or whether it occurred at the beginning 
of the project or a year later. A test score point, on 
the other hand, is likely to represent a different 
inCTiement of performance at different performance 
levels, and at different times. Thus, pre- and )pcst- 
test gaps that are equal in terms of test-i^core points 
may not be equal in terms of group performance 
levels. 



Mathematical adjustments 

are appropriate. These and other problems discussed below can be 

reduced to minimal proportions through mathe- 
matical manipulations. The required manipulations 
are performed automatically by the software that 
has been developed to accompany this Users' Guide, 
and you need not make any attempt to understand 
them unless you intend to do the analyses by hand 
or unless you find them intrinsically interesting* 
. Manual procedures along with the rationales that 
underlie them are described in Appendix H. 

Preparing data for 

analysis. Four simple steps are required to prepare the data 

for either computer or manual analysis. First, you 
must be sure that the data you enter into the 
analyses are either raw (number correct) or scale 



86 



scores . You may not use percentiles, stanines, 
NCES, or grade-equivalent scores. 

Scale scores are preferred to raw scores. Be sure to 
ask for them if you use a test-scoring service* If you 
are doing manual test scoring, however, it is prob- 
ably not worth converting individual-student raw 
scores to scale scores. 

All gap reduction calculations can be done using 
raw scores if the treatment and comparison groups 
took the same level of the test. If they were tested 
with different levels, you must work with scale 
scores-but you can convert mean raw scores to 
their scale-score equivalents. It is not necessary to 
convert individual-student scores. 

Students with missing 

data are excluded. Second, you must be sure that all students included 

in the analyses have both pre- and posttest scores. 
Students with only one or the other must be ex- 
cluded. This rule applies to both project and com- 
parison groups. 



^Scale scores are gir^en different names by different test publishers. These names 
include tandard soores, expanded standard scores. Achievement Development 
Scale Scores, converted scores, and Growth-Value Scores. All of them refer to spe- 
cially constructed scales that span the various levels of a particular test and thus 
provide a vehicle for converting out-of-level raw scores to in-level percentiles. 
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Part-year participants 
require special analysis. 



Third, if any students were pretested late (because 
they entered the project after the regular pretesting 
had been completed) or posttested early (because 
they left the project befoie the regular posttesting) 
you must note the months when they were actually 
tested. If you use the computer program, it will ask 
for this information. If you undertake a manual 
analysis, you will have to use this information to 
calculate extrapolated test scores (see Appendix !)♦ 



Gap reduction with 
normative data. 



Finally (but only if you use norms as your com- 
parison group), you must find the raw or scale 
scores that correspond to the 16th, the 50th, and the 
84th percentiles (for the appropriate grade levels) 
at both pre- and posttest times. This final data- 
preparation step is best explained by means of a 
concrete example. 



Suppose you are working with a group of third- 
grade LEP students who were pretested in May of 
their second-grade year and posttested in May of 
their third-grade year. You must find the norms 
tables for the level(s) of the test you used* You 
must then find the raw- or scale-score-to-percentile 
conversion table that is appropriate for May of 
second grade (grade 2.8) and the one for May of 
third grade (grade 3.8)- 

Note: Test publishers will provide either a raw*score- 
to-percentile table or a scale-score^to-percentile table^ 
not both. If you have a scaIe*score-to-per?entile table, 
but want the raw scores corresponding tv> the l6th, 
50th) and 84th percentiles, you will have to And the cor * 



ERLC 



88 



Using the computer 
prt^ 'am. 



responding scale scores first and then convert them to 
raw scores using a separate raw-score-to-scale-score 
conversion table. You will have to follow a similar two- 
step process if you want scale scores but the publisher 
only provides a raw-score-to-percentile conversion 
table. 



If you are using the computer program to do the 
gap-reduction calculations, you will simply enter the 
values you found in the norms table when the com- 
puter asks for them. If you are using manual pro- 
cedures, step-by-step instructions are given in Ap- 
pendix H. 



In essence, the computer program and the Appen- 
dix H procedures "standardize" the four test-score 
data points (mean pre- and posttest scores for the 
project and comparison groups) so that scaling 
problems are minimized and scores from different 
tests are made comparable. It also calculates the 
amount of gap reduction fiom the standardized 
data points and generates another measure called 
the Relative Growth Index (RGI). 



Understanding Relative 
Growth Indices (RGIs). 



RGIs express, in percentage terms, the amount by 
which the progress of the project group exceeded or 
fell short of the progress of the comparison group. 
An RGI of +20% means that the progress of the 
project group was 20% larger than that of the com- 
parison group. An RGI of -8% means that the 
progress of the project group was 8% less than the 
progress of the comparison group. 
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RGIs have one significant advantage over gap- 
reduction measures-they are independent of the 
heterogeneity of the comparison groups. What this 
means is that RGIs can be meaningfully compared 
between projects that use different types of com- 
parison groups (notably grade mates versus norms). 
Gap-reduction measures will tend to be larger for 
grade-mate comparison groups than for norms be- 
cause local groups tend to be less diverse than na- 
tional samples-but RGIs will be unaffected. 

When not to 

calculate RGIs. RGIs also have one significant disadvantage when 

compared to gap-reduction measures. If the com- 
parison group makes no progress or negative 
progress, the index becomes meaningless. RGIs 
should thus only be used in conjunction with 
achievement test scores (where posttest scores will 
almost certainly be higher than pretest scores for 
both the project and the comparison group). They 
should not be used when no "growth" is expected of 
the comparison group, as would be the case with 
indicators such as absenteeism rates or classroom 
grades. 

When no growth is expected to occur in the com- 
parison group, your analysis should end with gap- 
reduction calculations. 
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Using Medians Rather than Means 



Medians may be used 
instead of means. 



Medians are preferable 
under some conditions. 



Means and medians are both statistics that sum- 
marize the performance levels of 3roups of stu- 
dents. If score distributions are symmetricalp the 
median will be approximately equal to the mean. 
Such equality is typical of the score distributions of 
ncrming samples on standardized achievement 
tests. Indeed, it is typical of the score distributions 
of most groups on well made tests that were neither 
too easy nor too difficult for them (see Chapter V). 



In biLmgiial education projects, it is not uncommon 
for tests to be used that are too difficult for project- 
group students. When this occurs, the scores of 
those students are prevented from being as low as 
they should be. The result is that the mean of the 
group is artificially inflated. 



While the group's mean score can be affected if 
there is even one student who does not know the 
answer to a single question, the group's median will 
not be affected unless there are many such students 
(theoretically up to half of those tested). Thus, if 
there is evidence that some students-but fewer 
than fifty percent of them-did not know the 
answers to any of the questions, the median 
provides an estimate of where the mean would have 
been had the test been able to measure lower per- 
formance levels accurately. Using . median in 
your gap-reduction calculations would thus be more 
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appropriate than using the (spuriously inflated) 
mean. 



Too difficult tests 
produce biased means. 



The problem just described is most likely to be en- 
countered by the project group at pretest time-and 
you should check that possibility carefully. The 
problem is less likely to occur at posttest time, but 
you should still check. It is also possible that the 
test may be too easy for the comparison group- 
particularly at posttest time. Whenever tests are 
too easy or too difficult, group medians should be 
used rather than group means. Otherwise, means 
are preferable to medians because they are mor^ 
stable. 



Software substitution 
of medians for means. 



The computer program for this system automati- 
cally substitutes the median score of the treatment 
group for its mean score whenever the median 
score is one-fifth of a comparison-group standard 
deviation lower than the mean score. If you per- 
form the gap-reduction computations manually, you 
should follow the same procedure. 



It is quite acceptable to mix means and medians in 
a single analysis (although this practice would make 
it virtually impossible to determine the statistical 
significance of the findings). 
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Dealing with Regression Biases 



Whenever a subgroup of individuals is selected 
from a larger group because they scored above or 
below some criterion (cutoff) value, their mean 
score on any subsequent testing will be closer to the 
mean of the original group than it was on the selec- 
tion test. This relationship will bold regardless of 
whether the selected subgroup is retested im- 
mediately after their selection or at some later date. 
This phenomenon is called statistical regression, or 
regression to the mean. 

Regression biases can 
contaminate growth 

estimates. hi bilingual education (where project participants 

are usually selected on the basis of low test scores), 
regression to the mean causes scores of the selected 
students to be higher on subsequent testings than 
they were on the selection test. Thus, if the selec- 
tion test is also the pretest and the subsequent test 
is the posttest, re will be a pre-to-posttest gain 
that results solely from statistical regression, Cai'e 
must be taken to recognize this gain as spurious and 
not mistakenly attribute it to the project, 

Factors that ajfect the 

size of regression biases. The amount of regression-effect bias that will occur 

when a set of test scores is used for both selection 
and pretest purposes depends on two factors: (a) 
the difference between the mean score of the 
selected subgroup and the mean score of the total 
group (the greater this distance, the more regres- 
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sion will occur) and (b) the correlation between the 
selection-pretest and the posttest (the lower the 
correlation, the more regression will occur). 

When regression may 

be a problem This type of pretest-based selection is likely to oc- 

cur in many-perhaps most-Title VII projects. 
Consider the following scenario: 



Students are selected for project participation on the 
basis of a home language survey and a subsequent 
English language proficiency :est Students scoring 
below a pre-established cutoff score are admitted to 
the project, while those scoring above the cutoff are 
not. 



How to avoid 

regression biases. The issue now becomes whether or not the scores 

obtained by the selected students will be used as 
their pretest scores for the evaluation. If not-in 
other words, if they are subsequently administered 
a separate pretest-there is no need to introduce a 
correction for the regression-effect bias.^ If the 
scores students obtain on the English language 
proficiency test are also used as their pretest scores, 
a substantial amount of regression will occur be- 
tween the selection-pretest and the posttest. You 
must correct for it. 



^There will be a small amount of regression-effect bias even with a separate pretest. 
We regard it as too small to worry about, however, even though it could be statisti- 
cally removed. 
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Before discussing the method of correcting for the 
regression-effect bias, it is useful to extend the 
scenario begun above and examine another sima- 
tion that requires correction for regression. 



. another problematic 
situation. 



After one year of participation in the bilingual project, 
students are posttestecL Assuming an annual testing 
cycle, the posttest scores for the first year will also 
serve as the pretest scores for the students who remain 
in the project for a second year. Some students may 
have achieved the exit criterion^ however, and may 
leave the program. 



^f the year 1 posttest scores are u*^ed in any way to 
determine who should be exited from the project, 
you again have a situation in which (year 2) pretest 
scores were used to select the students who will 
remain in the project. Regression to the mean is 
the inevitable consequence, and you must correct 
for it in order to avoid biased growth estimates. Of 
course, you always have the option of avoiding the 
problem by not using selection test scores as pretest 
measures and by not using posttest scores in any 
way when you make exit decisions. 



Automatic correction for 
regression biases. 



The software that accompanies this Users' Guide 
automatically corrects for regression biases. 
Manual procedures are described in Appendix J. 
You should note, however, that the correction pro- 
cedure assumes that selection into the project is 
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based solely on the test scores that are subsequently 
used as the pretest measures. 

If selection-test scores are combined with other 
measures to form a composite pretest, the correc- 
tion procedures will overcorrect. Similarly, the 
statistical adjustment will be appropriate only If 
posttest scores are the sole criterion for exiting stu- 
dents from the program. If teacher judgments, 
classroom grades, or other considerations affect the 
existing decision, the adjustment will be excessive. 

Bracketing the 

regression bias. if the regression-effect adjustment is going to be ex- 

cessive for one of the reasons just discussed, we 
recommend that you calculate gap reductions and 
RGIs both with and without the adjustment (the 
computer program does this for you). The "correct" 
values will then be bracketed between values that 
will be somewhat too low (those with the 
adjustment) and values that will be somewhat too 
high (those without the adjustment). Depending on 
the relative importance of test scores and other fac- 
tors in the entry/exit decisions, you may be able to 
estimate approximately where the true values will 
fall within the bracketed range. 

If posttest scores from one year are used both to 
help decide which students will be exited and as 
pretest scores for the non-exited students, the scores 
of the exited students will be needed for computing the 
regression-effect adjustment. Identifying these scores 
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thus becomes a fifth step in preparing your test data 
for analysis (see pp. 89-90). 



Assuring Representativeness of the Data 



Evaluation findings should 
apply to all students 
served 



The current evaluation regulations specify that the 
findings of your evaluation must apply to all stu- 
dents served by the project* This requirement 
means that, if your project experiences high student 
turnover, you will need to make some provision for 
pretesting students who enter after the "regular" 
pretesting has been completed and/or posttesting 
students who leave the project before the "regular*' 
posttesting is begun* Any student who has par- 
ticipated in the project for a minimum of 100 days 
should be considered as having been served, and all 
such students should be included in the evaluation. 



Test scores of part-year 
participants require 
extrapolation. 



Students who were served but who participated in 
the project for less than the full year will, 
presumably, have made less progress than full-year 
participants. Thus pooling the data from part- and 
full-year participants will tend to make the project 
appear less effective than it really was unless the 
data from the part-year participants are extrapo- 
lated to provide estimates of what their gains would 
have been with full-year participation. The com- 
puter software designed to accompany this Users' 
Guide performs such extrapolations automatically. 
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Manual procedures are described in Appendix 1. 



Separate analyses of 
full- and part-year 
students are advisable. 



Even when the data from part-year participants are 



extrapolated to yield full-year estimates, you should 
be aware that growth may not be constant over the 
year so that part-year participants may have grown 
faster or more slowly than full-year participants- 
depending on which part of the year they attended. 
Also, the hierarchical nature of some learning tasks 
may make it difficult for late enterers to benefit 
from the project at all. 

The preceding paragraph suggests several reasons 
why combining even the extrapolated data fi'om 
part-time participants with those for full-time par- 
ticipants may yield misleading results. What we 
suggest is that you analyze the data several ways. 
We reconraiend, for example, that you calculate 
separate RGIs for part-year and full-year par- 
ticipants. 



might also conduct separate analyses for late en- 
terers and early departers-or for shorter-term and 
longer-term participants. You should not place too 
much confidence in analyses based on fewer than 
about 10 to 15 students, however, since the RGIs of 
small groups will be unstable. Given that general 
guideline, experiment with whatever subgroupings 
you think might make sense* 



Still finer subgroupings 
may be desirable. 



If you have enough part-year participants, you 
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Going Beyond the Gap-Reduction Design 



The regulations do not require that you go beyond 
the gap-reduction design. But the gap-reduction 
design does not provide a quantitative estimate of 
the project's impact. Such an estimate can only be 
obtained through proper implementation of an ex- 
perimental or quasi-experimental design. 

Many projects may not be able to implement an ex- 
perimental or quasi-experimental design because 
the kind of non-project comparison groups they 
require may be excessively difficult or impossible to 
obtain. On the other hand, the value of your 
evaluation would be enhanced if you were abl« to 
obtain a sound estimate of project impact. 

With this objective in mind, we have described, in 
Appendix G, three quasi-experimental designs that 
you may be able to implement-depending on your 
particular situation. We encourage you to read 
Appendix G and, if you can, to implement one of 
the designs presented there. 
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VIIL PROCESSING AND ANALYZING DATA 



Two types of data 
must be dealt with: 
outcome and process. 



There are two basic types of data to be reduced and 
analyzed: outcome data and process data. 
Generally speaking, outcome data are linked to in- 
dividual students while process data are linked to 
various components or aspects of the project. In 
this chapter, we discuss data reduction and analysis 
for individual outcome and process measures. In 
Chapter IX we discuss integrating and interpreting 
the analytic findings. 



Analyzing Student Outcomes 



Outcome data include 
test scores and non-test 
data. 



Student outcome measures include both test scores 
and non-test data (e.g., number of days absent). 
We begin with the test data. 



Test scoring can be done either manually or by 
machine. We recommend machine scoring if you 
can afford it. The advantages pertain mostly to ac- 
curacy and can be sizeable, particularly if the scor- 
ing task includes the conversion of raw scores to 
scale scores. 
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Use a scoring service if 
you have large numbers 
of students. 



Check for accuracy of 
manual scoring. 



Determining rav/ (number correct) scores by hand 
may noi be excessively burdensome with small 
numbers of students. With large numbers of stu- 
dents, boredom, carelessness, and a desire to be 
done with the task often combine to produce high 
error rates. Quality control procedures are 
definitely in order. 



For each scorer, you should have a second person 
check the accuracy of a randomly selected 5% of 
the scores (e.g., for every 20 tests, rescore one). If 
any error is found, another 5% of the scores should 
be checked. If an error is found in this group, all 
the tests should be rescored. 



Use scale scores with 
functional level testing. 



As mentioned earlier, you must use scale scores if 
you do functional level testing. Without them you 
cannot quantify the gaps between groups that were 
administered different levels of the test. 



Convert the mean raw 
score to its scale-score 
equivalent. 



Unfortunately, converting raw scores to scale scores 
is an even more error-prone procedure than simple 
scoring when done manually. If you must use 
manual procedures, we recommend that you calcu- 
late mean raw scores for the various groups of in- 
terest and convert them to their scale score equiv- 
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Clean answer sheets 
before machine 
scoring. 



alents rather than converting individual-student aw 
scores. 

You will not get exactly the same result with the 
short-cut procedure as with the more correct but 
cumbersome and error-prone alternative, but it will 
usually be very close. With machine scoring you 
should always request that the scoring service 
provide you with individual-student scale scores. 



Whether you use machine or manual scoring pro- 
cedures, you will have to do some preparation of 
the test booklets or answer sheets. In both cases 
you need to verify that the student identification in- 
formation is compete and accurate. In the case of 
machine-scored answer sheets, you need to erase 
stray marks and make sure that each student's 
answers are marked clearly enough to be picked up. 
You will have to watch for other problems such as 
multiple answers to the same question, pattern 
responding, etc., if you do manual scoring. Machine 
scoring will pick them up automatically. 

Compiling non-test data should be a relatively 
straightforward matter, but it makes a substantial 
difference whether the individual student data you 
need will be stored in district or project files and 
whether those files will be computerized or kept on 
paper. If the files are computerized, you will be 
able to do substantially more analytic work with 
them than if they are kept on paper-particularly if 
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you can download data from the files directly into 
analytic routim^s^ Sorting paper files and entering 
data manually can be very time-consuming ac- 
tivities. 



Set up files for 

each student Regardless of whether data storage is com- 

puterized or not, you must set up files for each stu- 
dent in your project group (and possibly for each 
student in your comparison group). Each student's 
file should be labeled with his or her name and 
identification number and should include at least 
the minimal student characteristics information 
specified in Chapter IV of this Users' Guide. In ad- 
dition, all test scores used for evaluation should be 
included in raw-and/or scale-score form along with 
the testing dates and the names, forms, and levels of 
the instruments used. 



Include required 
non-test data in student 

files. Each student file should also include number of 

days absent (broken down into at least two or three 
calendar periods), grade retentions (if any), dis- 
ciplinary actions (if any), project entry and (if 
exited) exit dates, and referral to or placement in 
special education or gifted and talented programs. 
If the students have dropped out or entered 
postsecondary education institutions, those facts 
(along with their dates of occurrence) should also 
be included. Finally, it will be useful to include 
codes for the classrooms/teachers where or from 
whom students received project services. This in- 
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formation will be particularly asefiil if your data 
base is computerized. 

Steps for manual 

data entry. Coding and/or data entry. If data will be analyzed 

manually, you should: 

• copy individual student data onto sunraiary 
forms and sort the sunmiary forms by class and 
grade level (Remember, the correction for 
regression requires the posttest scores of last 
year's participants who were exited,) 

0 randomly check 5% of the entries as described 
above. 



0 make a copy of each sununary form and store 
the copies in a location different from where 
the originals are kept. 

Steps for computer- 

managed files. For computer processing, data summary forms can 

also be developed to make data entry easier. 
However, if the data are well organized and you 
have a data management program available (dBase 
in, for example), you should enter data directly into 
the computer to save time and to reduce errors. 



To check data entry, either 



• randomly check what has bein entered against 
the original records, or 



105 



112 



ERLC 



© enter a sample of the data a second time and 
use a computer program to find differences be- 
tween the two sets of records. 



Constructing a data base. If you will be using 
manual data processing, each student's data (e.g., 
answer sheets and/or questionnaires) should be 
kept in a separate folder and filed so as to be easily 
retrievable. 

Questions about 
constructing and using 
a computerized data 

base. If you use a computer to create your data base, you 

must answer several questions; 

a Will you use the "central" district computer or 
your own project or group computer? 

• If you are going to use the central district com- 
puter, how are you going to add the unique 
project data to the district's centralized data 
base to create the project data base? 

• If you are going to use your own computer to 
build your data base, will you be able to trans- 
fer some data from the centralized data base? 
If so, how? 

Centralized data bases 
may meet most of your 

information needs. Many districts keep centralized data bases on com- 

puters. Such records often include information on 
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students' backgrounds, grades, and test scores. 
Where such a data base exists, it may represent a 
single source for much of the data you need. 

If the district does have a data base, you should 
contact the person in charge of data processing for 
the district and find out if it would be possible to 
add your project data to the district data base. You 
should also explain the types of analyses you will be 
wanting to perform on the data. 

Transferring data 

between computers. If you decide to keep your project data on your own 

computer, but you also want to use the district's 
data base, careful planning can save you a lot of 
time and energy. First, you should find out: 

o whether your computer and the district's com- 
puter can conrniuiiicate with each other. 

0 whether the software on either computer will 
allow you to merge data files. 

d If both of these operations are possible, you can 
simply transfer the district's data to your com- 
puter and merge the data, or if necessary, trans- 
fer your data to the district computer, merge 
data, and transfer everything back. 

If neither of these operations is possible, then you 
may have to obtain printed copies of data from the 
^ district's computer and enter it into your computer 
file. 
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Six sound practices for 

data base management In setting up a data base, it is a good practice to: 



e Store data in different subfiles^ For example, 
there might be a student background data file 
and separate data files for each project year. If 
a student has the same ID number in each file, 
files can be merged as necessaiy for analysis. 

e Ensure the confidentiality of the data by storing 
them in a computer file under the secret access 
codes which are available for some data 
management programs. 

© Store multiple copies of files using different 
media (e.g., disk, tape, and hard copies). 

e Examine files periodically, especially when new 
data are added, and make any changes to en- 
hance their accessibility and safeguard against 
their misuse. 

• Use interactive data management software 
whenever possible. 

e Always include the district student ID number 
with each student's data. 
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The gap-reduction 
design meets all 
requirements for 
assessing academic 

progress. Analyzing the data. The regulations require that 

you evaluate the academic progress of (a) LEP 
project participants, (b) non-LEP project par- 
ticipants, and (c) former project participants now in 
mainstream classrooms. All three of these 
requirements can be met using the gap-reduction 
design (see Chapter VII). 

The regulations also require that, as appropriate, 
changes in student grade retention, dropout rates, 
' absenteeism, referral to or placement in special 
education classes, placement in programs for the 
gifted and talented, and enrollment in postsecon- 
dary education institutions be assessed and 
reported. 

Measuring changes in 
low-frequency events 
may require multi- 
year data. The baselines from which to measure change are 

presumably the pre-project rates. Unfortunately, 
some of the rates, such as placement in programs 
for the gifted and talented, are likely to be quite 
low. For this reason, it will probably not be pos- 
sible to obtain reliable measures of change from a 
Singh year's data. You will have to search histori- 
cal files from several years to determine pre-project 
rates and then compile data for several project 
years before any changes will be detectable. 
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Pre-project rates may 
be included in the 
needs assessment. 



Population shifts may 
invalidate before-after 
comparisons. 



Simple tabulations can 
be used to summarize 
project-related changes. 



A good needs assessment may have picked up on 
problems such as grade retentions and absenteeism. 
If statistics were compiled on those events and were 
included in the project application, they provide 
good baseline (pre-project) data. Statistics of the 
same type collected after a year or more of project 
operation should enable you to assess any changes 
that may have occurred. 



One problem you are likely to encounter is that 
school or district records may be inadequate or very 
difficult to access. A second problem is that the 
school population may have changed (perhaps due 
to an influx of refugee students or the creation of a 
new labor market). Under such circumstances, sys- 
tematic differences may exist between the pre- and 
post-intervention samples of students that would in- 
validate any attempts to assess project impact. You 
should then present the data and explain why 
whatever change was observed cannot be attributed 
to the project. 



Under more favorable conditions, we recommend 
compiling data on an annual or multiple-year basis 
for periods before and during project operation. 
The analysis, then, need be no more than a simple 
tabulation. You might, for example, report, "163 
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Contrast gains on 
curriculum-relevant 
and non-relevant 
items. 



(19%) of the 858 I^P chiidren enrolled in school 
were retained in grade during the three years im- 
mediately prior to the start of the project. Dming 
the first three years of the project operation, that 
total dropped to 101 (12%) of the 842 LEP students 
enrolled in school." 

If the sample size is sufficiently large (30 or more in 
each subgroup), you m-'-y want to mialyze the trends 
by grade, by ethnicity, or by other variables of 
interest. 



Secondary analyses can also be done to shed addi- 
tional light on the findings of the primary (e.g., gap- 
reduction) analyses. Assuming that you classified 
test items as curriculum-relevant and non- 
curriculum-relevant as part of the test selection 
process, you could have the two sets of items scored 
separately. You would then expect to see a larger 
percentage increase (from pre- to posttest) for the 
curriculum-relevant items than for the non-relevant 
items. 

Such a finding would tend to support the hypothesis 
of a positive project impact, while its opposite 
would suggest a negative impact. Evidence of this 
type can never be conclusive, however, since the 
non-relevant items might also be intrinsically more 
difficult than the relevant items or vice versa. 
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Ciassroom performance 
data can supplement 

other analyses. Analyses of classroom performance (curriculum- 

specific objectives) as reflected by average grades, 
curriculum unit^ completed, numbers of objectives 
mastered, etc., although not required by the regula- 
tions, could also be carried out. Such analyses 
might be particularly useful for identifying 
weaknesses in the project that need to be remedied 
(see below). They are also useful in confirming 
student outcome assessments derived from annual 
measurement of achievement growth. 

Qassroom grades and/or scores on teacher-made 
tests are likely to be available in most instances. If 
they are, it would be informative to calculate the 
correlation between classroom grades and posttest 
scores on your achievement test. (Your regional 
Evaluation ^^sistance Center can help you design 
such analyses). A positive correlation would 
provide direct confirmation that the test measured 
the same achievement dimension as classroom 
grades and indirect evidence that the test measured 
what was taught. 



District or statewide 
testing pro-ams can 
also provide useful 

data. Scores from district- or statewide testing programs 

are likely to have more desirable characteristics 
than classroom' grades, if the tests are not too difficult 
for project students. They could also provide useful 
confirmatory evidence if correlated with posttest 
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Discrepancies are the 
key to process 
evaluation. 



scores from the project's evaluation test. Further- 
more, if the district or state tests are administered 
on approximately the same schedule as the evalua- 
tion tests, you could correlate growth scores in addi- 
tion to postproject measures* Several such analyses 
may be possible if you have an efficient, com- 
puterized database. Otherwise you may have to 
limit your analyses to those required by the 
regulation.. 

Analyzing Project Processes 



Process analysis is largely a matter of looking for 
discrepancies between the project as intended and 
the project as implemented In some cases it may 
also be possible to quantify the magnitude and 
presumed importance of the observed dis- 
crepancies. Ultimately, the goal is to relate process 
discrepancies to possible discrepancies between an- 
ticipated and observed student outcomes for the 
purpose of effecting project improvement (see 
Chapter IX). 

Process discrepancies can affect the total project 
(e.g., instructional materials arriving three months 
late) or individual classrooms (e.g., two of the 
project teachers speaking English with a heavy 
accent). 
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Process evaluation is 
not required by the 
regulations. 



Discrepancies need to 
be quantified 



Discrepancies need to 
be quantified 



The regulations do not require any form of piocess 
analysis (the topic of the remainder of this chapter) 
nor any analyses that examine relationships be- 
tween processes and outcomes (the subject of 
Chapter IX). The main advantages to be gained 
from evaluation, however, relate to project im- 
provement. For this reason we urge you to under- 
take at least some of the analyses described in the 
remainder of this chapter and in Chapter IX. 



All forms of process evaluation will benefit from 
quantification of the discrepancies between what 
was intended and what actually happened-but the 
need is greater and the task more difficult when 
there are multiple project classrooms per grade. In 
this situation, some sort of scale must be developed 
to reflect the differences that exist among them. 



Scaling is no problem for discrepancies such as per- 
cent instructional usage of LI, or ratio of positively 
to negatively reinforcing teacher utterances. These 
discrepancies are basically self-scaling. Others, 
such as heaviness of teacher accent or teacher ex- 
pectations regarding student achievement levels, 
however, can be difficult to quantify. 



Avoid vague constructs. 
Instead, count 
observable events. 



Rank teachers/class- 
rooms rather than rate 
them. 



The amount of difficulty you are likely to ex- 
perience in quantifying discrepancies will be 
directly related to the vagueness of the relevant 
constructs. We have already suggested (see Chap- 
ter IV) that complex constructs require high- 
inference observations and that replacing them with 
more directly observable (lower inference) sub- 
stitutes will yield significant interrater-reliability 
benefits. Parallel benefits will accrue with respect 
to the quantification of discrepancies. Replacing 
"heaviness of accent" with "number of words 
mispronounced" (wrong sound or wrong syllable 
emphasized), for example, changes a high-inference 
rating task into a low-inference counting task. Not 
only will interrater reliability increase-so will the 
adequacy of quantification. 



If there appears to be no adequate, directly observ- 
able substitute for a complex conceptual construct 
(e.g., level of teacher expectations), our next 
recommendation is rank-ordering classrooms/ 
teachers on the characteristics of interest rather 
than rating them. While ranking ignores the size of 
differences between the entities being ranked, 
rating tends to distort them. And if you use mul- 
tiple observers (which you should do anyway to 
assess th3 reliability of your data) average rankings 
will recapture some information relative to the size 
of differences. 



ERLC 



115 

122 



Try to make your data 
collection non- 
threatening. Either rating or ranking teachers can be 

problematic due to teacher-union rules or other 
political constraints. This is another reason for at- 
tempting to make your process data as objective as 
possible. Counting frequencies of behaviors is less 
threatening than evaluating teacher performance 
on some dimensions. In any case, you need to make 
it clear that you are documenting project processes, 
not evaluating teacher perfonnance. Then, even if 
you are denied the opportunity to conduct class- 
room observations, you may be able to obtain 
reasonably accurate data by means of teacher inter- 
views or questionnaires. 

Once you have quantified discrepancies for 
classrooms/teachers, you can calculate mean or 
median values for each grade level served by the 
project. You will want to have classroom-level data 
for within-year analyses and project-level data for 
between-year analyses. This distinction and how 
the different types of analyses can inform you about 
process-outcome relationships will be the subjects 
of the next chapter. Before looking at these causal 
relationships, however, it may be useful to examine 
process data in somewhat greater depth. 

Combine 

discrepancy scores to 
create a project 

implementation index. There are several things you could do that would 

simplify analyses of process-outcome relationships 
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and perhaps make them easier to interpret. First, 
you could combine discrepancy scores across 
processes and thus obtain an overall project- 
implementation score for each classroom. 

A simple, but often less than ideal, approach is 
simply to sum the individual discrepancy scores. 
The problem with this approach is that it will 
automatically weight each component of the com- 
posite score according to its variability across 
projects. Thus, if there are large differences among 
classrooms with respect to a particular process, that 
process will be more heavily weighted in the com- 
posite scores than a process that is less variable 
across classrooms. 

Standardize 
discrepancy scores to 

weight them equally. The recommended procedure for dealing with this 

problem is to calculate the standard deviation of 
discrepancy scores for each process across all class- 
rooms and then divide each individual score (for 
the same process) by that standard deviation* 
Summarizing the "standardized" discrepancy scores 
for each classroom will then give you composite 
implementation scores in which each process is ap- 
proximately equally weighted. 

You may stop at this point, or you may wish to 
weight the scores according to your assessment of 
their importance. If you wanted to give twice as 
much weight to instructional usage of LI as to 
pronunciation of English words, for example, you 
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Block teachers/class- . 
rooms into analytic 
groups. 



would simply multiply each classroom's standard- 
ized discrepancy score for instructional usage of LI 
by 2. To obtain the properly weighted composite 
score, you would then sum the weighted, standard- 
ized discrepanqr scores for each classroom. 



A second procedure that will be useful for basic 
analyses of process-outcome relationships is called 
"blocking." Instead of working with individual class- 
room scores, you could sort (block) the classrooms 
into high, medium, and low groups. 

The blocking process will serve the purpose of in- 
creasing the stability of your data. The mean score 
of a group of classrooms will be more stable than 
the scores of the individual classrooms. The in- 
creased stability will have the effect of making your 
analyses more sensitive to real relationships be- 
tween process and outcome variables. This topic is 
discussed more fully in Chapter IX. 



ERIC 



118 



IX- INTEGRATING AND INTERPRETING RESULTS 



Begin by examining your 
outcome results. 



If you have carried out all the procedures recom- 
mended in the previous chapters of the Users' 
Guide, you will have collected and analyzed two 
types of data: process data and outcome data. Al- 
though you will have looked at outcome test results 
for the project as a whole, you need to examine 
them carefully by grade level and possibly by class- 
room. Deciding whether they exceed, meet, or fall 
short of your expectations should be the first step in 
data interpretation. 



Interpreting Gap Reductions and RGIs 



Disappointing outcomes 
need an explanation. 



Perhaps you are perfectly satisfied with your out- 
comes, but often some or all of them will be disap- 
pointing. If your gap reductions and RGIs are 
lower than you expected them to be, you will want 
to find out why. You can address this question by 
proceeding through the steps outlined in this 
chapter. 



Positive outcomes also 
require interpretation. 



Even if your gap reductions and RGIs are better 
than expected, you should not skip the step of data 
interpretation. You need to integrate and interpret 
your data in order to establish causal linkages be- 
tween project processes and student outcomes. 
High RGIs may not have resulted from the project. 
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Consider the possibility that they are high because 
students became much more test wise between the 
pretest and posttest, for example. If this were 
indeed the case, you should expect smaller RGIs in 
following years, even if the project is working as 
planned. As another example, suppose a high RGI 
is due to comparison group students being post- 
tested with too easy an instrument. If in a following 
year, a higher level of the test is used, project out- 
comes will appear to be less satisfactory. Thus, it is 
never enough to say that your project met its out- 
come goals. You must also be able to suggest 
reasons for its success, and to show evidence that it 
was the project, and not some unrelated events, that 
produced the outcomes. 



Compare RGIs to other 
outcome data. 



Integrating test data with other ^^utcome data. The 
first step in interpreting RGIs is to look at the other 
outcome data you have collected. These may be 
data on absenteeism, scores on teacher-developed 
tests, or mainstream classroom grades of former 
participants. Do these additional data seem to con- 
firm the RGIs, or call them into question? If all 
your outcome data point in the same direction, you 
can dismiss some of your doubts about the validity 
of your test score findings. If outcome data conflict, 
however, further investigation is necessary. 



Conflicting outcome data 
call for detective work 



If there are conflicts between other outcome data 
and standardized achievement test scores, consider 
the possibility that the standardized test is invalid 
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for your project. If you did not go through the test 
selection steps presented in Chapter V of the Users' 
Guide, do so now. Your RGIs may be a function of 
poor test selection and may not reflect project 
impact. 

Look for patterns 

in outcome data. Perhaps only some of your other outcome data con- 

flict with your RGIs. For example, suppose your 
RGIs in English language arts are low, but at some 
grade levels students* scores on published 
curriculum-based tests have been quite satisfactory. 
This pattern of results suggests both that the con- 
tent and validity of the achievement test was low (it 
was insensitive to performance differences that the 
curriculum-based tests picked up) and that the lan- 
guage arts component of the project was more ef- 
fective at some grade levels than at others. 

If absenteeism continued to be high at some grade 
levels, but improved at others, it might be logical to 
infer that affective components of the project were 
better implemented at the latter grade levels than 
at the former. 



Organizing data by 

student groupings. You should always group your data by grade level. 

Then ask, "are some grade levels consistently lower 
or higher than others?" You should also group data 
by classrooms or groups of classrooms and ask, "are 
there noticeable differences between classrooms?" 
If your students are heterogeneous, you might try 
grouping them by factors such as home language or 



ERIC 



121128 



parent literacy. Perhaps your project has been 
successful with some groups and not with others. 



Always consider the 
possibility that your data 
may be invalid 



If your outcome data are mixed and show no dis- 
cernible pattern, you are in the unfortunate position 
of having few conclusions to draw about project 
outcomes. Under such circumstances, you should 
re-evaluate the validity of all your outcome 
measures 1^ reviewing the following questions: 



Was the test used to obtain RGIs valid and ap- 
propriate for the project and comparison 
groups? 



9 Were all outcome data collected in a careful, 
consistent manner? Were there irregularities 
in test administrations? 



9 Were computational errors possible in data 
analysis? 

• Were test scoring errors possible? 

0 Did you use the correct conversion and norms 
tables in your computations? 

What caused student 

outcomes? Integrating outcome data with process data. To in- 

terpret your outcome data, you need to identify the 
most likely cause of the outcomes you obtained. If 
you obtained high RGIs, but had a minimal treat- 
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Review your process 
data for surprises. 



ment (e.g , 10 minutes of ESL per week), you can 
assume that the project did not cause the high 
RGIs. You should try to find out what did. If you 
had a well implemented, comprehensive treatment 
but low RGI's, you should also try to determine the 
reason. Whatever your outcome data look like, you 
owe it to everyone involved in the project to try to 
determine: 

• what project features, if any, most likely con- 
tributed to the outcomes. 

• what project features did not appear to affect 
the outcomes. 

9 what nonproject features played a role in the 
outcomes. 

You should address these questions by examining 
your process data together with your outcome data. 

As a first step, look over all your process data- 
teacher, project, and student characteristics; im- 
plementation data; and non-project events. Are 
there any unexpected elements in the data? For 
example: 

• Were teachers less qualified than the project 
design called for? 

• Did the project serve sti- ients from unexpected 
ethnic or language groups? 
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• Did the project serve more (or fewer) students 
than expected? 

• Did classroom observations reveal failures in 
implementation or changes to the project 
design? 

• Was the project allocated necessary space and 
equipment? 

e Was parent involvement as extensive as 
planned? 

• Were necessary materials available? 

• Did teachers attend planned preservice and in- 
service sessions? 



Higher implementation 
should lead to higher 
outcomes. 



• Could other specific events in the school or 
community have affected student outcomes? 



Once you have identified areas in which the project 
did not operate as planned, you should review re- 
lated outcome data. For example, if second-grade 
teachers received the lowest implementation scores 
of any grade level observed, you would expect to 
see poorer project outcomes at that grade leveL 
Conversely, if fifth grade teachers had the highest 
implementation scores, you would expect higher 
outcome scores at that grade level. If both situa- 
tions pertained, you would certainly expect fifth 
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Project emphasis should 
be reflected in outcomes, 
but surprises occur. 



grade outcomes to be better than second grade out- 
comes. If this is the case, it suggests that when the 
project is properly implemented, students achieve 
more. 

On the other hand, process discrepancies may not 
reduce project impact and may even enhance it. 
Placing bilingual paraprofessionals in classrooms 
taught by teachers who are without bilingual 
credentials couldy for example, prove to be a more 
effective strategy than the originally intended use of 
credentialed bilingual teachers without aides. 



Sometimes your project may have unexpected out- 
comes. Suppose that your project emphasized basic 
skills instruction heavily, but was not at all prescrip- 
tive about how science and social studies should be 
taught. Your outcome data show that students 
made a very impressive jump in social studies-a 
much better outcome than in reading or math. You 
should consider the following questions: 

9 If the social studies test is in English, did it 
serve as an English reading test? In that case, 
students' scores may reflect increased English 
proficiency. 

• Did the social studies test have higher content 
validity than the other tests? 
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• How was social studies taught? Did teachers 
use techniques that should be considered for 
other instruction? 



Patterns of outcome 
and process data. 



Display your data 
graphically 



Where process data show a significant discrepanqr 
from the project design, you would expect to see 
lower outcomes. This could happen in specific 
classrooms, schools, subject areas, grade levels, lan- 
guage groups, or years. You will need to group stu- 
dents in ways suggested by your process data and 
calculate RGIs to see what the pattern is. An effec- 
tive project should produce lower RGIs at points of 
known failure or weakness. The pattern will not 
necessarily be completely consistent, but it should 
be suggestive. 



The simplest way of examining patterns is to make 
a graphic display of your data. As an example, sup- 
pose that you have overall implementation scores 
for every classroom in your project and you also 
have classroom-level RGIs. You could use a bar 
graph such as that shown in Figure 4 to illustrate 
the relationship (which in this instance is quite 
strongly positive). 



The situation can easily be more complicated, 
however. Some project components may not be ef- 
fective with your project group. Failures to imple- 
ment them properly may not show up as reduced 
outcomes. Whatever substituted for proper im- 
plementation may even have produced better out- 
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comes. You need to be alert to this type of unex- 
pected relationship, so that you can avoid taking in- 
appropriate corrective actions. 




1 2 3 4 5 6 7 

Classrooms 

Figure 4. Graphic display of outcome and process data. 



There may be 

no patterns. Perhaps there is no discernible relationship be- 

' tween process and outcome data. Consider these 
questions: 
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• Are the process and outcome data valid? 



• Was the project as implemented substantial 
enough that one could reasonable expect it to 
make a difference? 



• Were student needs accurately defined? 

• Was-the project designed to meet those needs? 

• ■ Were the outcome measures designed to 

measure whether those needs were met? 

Drawing Conclusions and Making 
Recommendations 



The ultimate purposes of your project evaluations 
are to enable you to draw conclusions about project 
impact and to make recommendations about 
project improvement. Your conclusions and 
recommendations should derive from your integra- 
tion of process and outcome data. Outcome data 
alone, do not demonstrate a specific cause. Causal 
relationships are only credible when the linkages 
between processes and outcomes both make sense 
and are supported by the data. 

Show evidence that the 
project caused the 
outcomes. 



In drawing conclusions, begin with any positive out- 
comes. Can you show evidence that the project 
contributed to these outcomes? If not, is there 
evidence to suggest other causes? In the light of 
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these findings, whac should the project do next year 
(i.e., modify the project, modify the evaluation, or 
continue as planned). 



Next proceed to disappointing outcomes. Do these 
appear to be related to specific project weaknesses? 
If so, how can the project be s»:rengthened? Are the 
outcomes possibly related to probleifis in 'he 
evaluation? Can these be corrected in succeeding 
years? If the results are uninterpretable, how can 
they be made more interpretable in the future? 

Were your expectations 

too high or too low? Finally, look at your expectations for outcomes in 

the light of what you now know about your project 
and your students. Were your expectations too 
high? If your project is necessarily a very limited 
one, your results will probably be limited also. 
High student mobility, for example, will dilute the 
effects of your project, and there is not much that 
can be done about it. 



Perhaps your expectations were too low. If the 
project met or exceeded all your expectations, it 
may be appropriate to raise your sights in following 
years. 

Confer with "^'^"^ earn 
before finalizing your 

recommendations. Conclusions and recommendations should be dis- 

cussed with your evaluation team, if at all possible. 
They may have insight into project weaknesses 
and/or sources of inaccuracies in evaluation data. 
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X. REPORTING 



Your report must 
address all points 
specified in the 

regulations. The regulations require that evaluation reports in- 

clude specific information about the students 
served, the services provided, the project staff, and 
the impact of the project on the targeted students. 
All of these requirements have been discussed at 
appropriate places throughout this Users' Guide. To 
refresh your memory we also provide a checklist of 
those information items later in this chapter. 

Prepare your report to 

satisfy all your audiences. Obviously your evaluation reports must contain all 

of the information required by the regulations. 
Presumably, however, the Federal government will 
not be your only audience. You will have to cover 
the information needs of your other audiences as 
well-unless, of course, you are able to prepare 
separate reports for different audiences. 

Since your audiences are likely to range all the way 
from policy makers (school board members, school 
and district administrators), who will have no inter- 
est in details, to educational researchers who will be 
concerned with the soundness of methodologies you 
employed and the credibility of the results you ob- 
tained, the organization and presentation of your 
report deserve careful consideration. 
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We recommend that you begin your report with 
some sort of non-technical summary for administra- 
tive personnel. Details describing the implementa- 
tion of the evaluation and the processing of data 
should be presented later (perhaps in technical 
appendices) to satisfy researchers that all threats to 
validity were dealt with adequately and that the 
findings do not reflect large and/or unacknowl- 
edged biases. 

In preparing the report, remember to avoid techni- 
cal terms as much as possible. Use the active voice, 
and don't be afraid to use *\ve." A visually appeal- 
ing format and graphic presentation of data make a 
report easier to understand. Finally, when possible, 
make a verbal presentation of the results to inter- 
ested audiences. 

Executive Summary 
Begin by listing 
major findings. Keep 

it short. You should begin your "Executive Summary" with 

a very brief description of the project being 
evaluated, followed by a presentation of the major 
evaluation findings. Findings should be described 
in order of their importance (your readers may not 
get past the first few— especially if they seem 
unimportant) and as briefly as possible (if they are 
brief enough your audience might even read all of 
them). 



Follow a few basic 
reporting guidelines. 
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Start with positive 
findings. 



Include a brief summary 
of earlier years' findings. 



Although your discussion of findings will be limited 
to the major ones, it is desirable to mention that 
there are additional findings. You can then 
reference the location(s) in the main body of the 
report where they are discussed. 

It is usually a good idea to present positive findmgs 
first, and if reasonable, to describe negative findings 
as "areas for project improvement or some similar 
euphemism. 



K your report deals with the second or subsequent 
year of a project operation you should briefly 
describe the findings of earlier years, changes made 
to the project as a result of earlier evaluations, and 
any improvements in outcomes that may have 
resulted firom those changes. No further informa- 
tion need be included in your summary, which will 
be most effective if you can keep its total length 
within five pages. 



The Main Body of the Report 

Your Introduction should 

cover seven topics. The main body of the report should begin with an 

Introduction that includes the following 
information: 

• A brief description of the community, district, 
and school settings (1 page). 
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0 A description of student characteristics (1 or 2 
pages). 

• A presentation of student needs assessment 
findings (1 page). 

e A narrative description of the treatment. This 
description should mention whatever dif- 
ferences there were between actual and in- 
tended treatment characteristics, but without 
excessive detail (3 pages). 

« A brief description of staffing and staff 
qualifications (1/2 page). 

« A brief description of materials and materials- 
development activities (1 page). 

a A brief description of parent and community 
involvement activitie.s (1-2 pages). 

Cover four topics in 

the Methodology chapter The second chapter of the report should describe 

the evaluation's Methodology. It should: 

• List the project's outcome objectives and 
describe how instruments and procedures were 
selected/developed to assess each of them. 

if Describe how and by whom both process and 
outcome data were collected. 
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• Summarize the data processing and analysis ac- 
tivities that were undertaken. 



• Indicate what steps were taken to avert threats 
to the evaluation's validity (in lay person's 
language-technical detail should be relegated 
to an appendix). 



Organize findings by 
objective. 



The next chapter should report the evaluation's 
Findings. This chapter should be organized by ob- 
jective. Fmdings should be broken down by grade 
level within each objective and by student group 
(LEP participants, non-LEP participants, and exited 
former LEP students currently in mainstream 
classrooms) within grade. 



A discussion of the quality of the data supporting 
each fi'iding should be integrated into the presenta- 
tion. Again, however, excessive detail should be 
relegated to an appendix. 



End with a Conclusions 
and Recommendations 
chapter. 



You may include a Discussion chapter if you feel it 
is warranted. Otherwise you may go directly to the 
report's final chapter. Conclusions and Recommen- 
dations. In this chapter you should attempt to in- 
tegrate data from the Findings chapter to address 
such broad questions as: 



d How well are project participants progressing 
academically? 
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6 What evidence is there that they are benefiting 
from project services? 

e Are the findings consistent with theoretical 
expectations? 

e Are observed outcomes better or worse than 
expected? 

• What project components appear to be most 
effective? 

• What project components appear to be least 
effective? 

o What changes could be made to the project 
that would (presumably) enhance its 
effectiveness? 

• What differences are there between this year 
and last year? 

• What changes might be expected next year? 

Don't be afraid to 

speculate. In most cases, your attempts to address these ques- 

tions will require speculation. You should always 
ind*::ate the degree of confidence you have in your 
hypotheses and the extent to which the data tend to 
support them and why. E ■) not hesitate to present 
alternative hypotheses along with the evidence sup- 
porting each. 
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If possible, provide a step-by-step guide for each 
recommendation you make, suggesting how it 
should be implemented. Concrete, specific steps 
are more likely to be carried out than general 
suggestions. 

Strive for objectivity. Excessively optimistic inter- 
pretations will cast doubts on the credibility of the 
entire report* On the other hand, informed 
speculation on your part can be helpful to the 
reader- Be sure to present the data that led to your 
speculation so your readers can draw their own 
conclusions. 

Checklist of Mandated Reporting Require- 
ments 

Be sure your report covers 

the mandated topics. The current (1986) regulations governing bilingual 

education evaluation are explicit about certain 
reporting requirements. These requirements 
include: 

Characteristics of the students served 
• educational background, 
a assessed needs. 



Describe specific 
procedures for 
implementing your 
recommendations. 
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© academic competencies in English language 
proficiency, native or other language 
proficiency (for projects of developmental 
bilingual education), and other academic 
subjects. 

Characteristics of the treatment 

9 specific educational activities undertaken. 

e pedagogical materials, methods, and techniques 
used. 

« relative amounts of instructional time spent 
with stuaents on specific tasks. 

Characteristics of the project staff 

0 professional qualifications. 

0 language competencies. 

Student outcomes 

The following information must be reported 
separately for (a) LEP project participants, (b) 
project participants whose primary language is 
English, and (c) former LEP participants currently 
in mainstream classrooms: 

• amount of time students received instructional 
services in the project or in another instruc- 
tional setting. 
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9 progress (as reflected by test scores) students 
have made in English language proficiency, LI 
proficiency (for projects of developmental 
bilingual education), and other academic 
subjects. 

• changes in the rate of student (a) grade reten- 
tion, (b) dropout, (c) absenteeism, (d) referral 
to or placement in special education programs, 
(e) placement in gifted and talented programs, 
and (f) enrollment in postsecondary educa- 
tional institutions* 

Also implicit in the regulations is the need to 
provide a description of the evaluation methodol- 
ogy, including particularly: 

• steps that were taken to assure that the evalua- 
tion findings would be representative, i.e., could 
be generalized to the entire population served. 

• steps that were taken to assure that the data 
collection instruments and procedures were 
reliable and valid. 

0 steps that were taken to assure that evaluation 
procedures minimized error (averted threats to 
validity). 

These implicit requirements should be addressed in 
the Methodology chapter. 
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You should contact your local Evaluation Assis- 
tance Center if you need help with any of these 
reporting requirements. 

Good luck! 
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Classroom Observation 



Observations can be conducted informally or formally* Informal observation 
is not guided by any predetermined scheme and the data or reportc it produces are 
based primarily on subjective judgment Formal observation is structured following 
specific rules for observing certain behaviors. Three generally recognized formal 
observation procedures-behavioral checklists, coded behavior records, and delayed 
report instruments-are discussed in this appendix. 

Formal observational techniques are most appropriate for documenting in- 
structional activities, verifying self-reports, and for identifying discrepancies between 
intended and actual program implementation. They can also be used to evaluate 
the extent to which the teachers upply in the classroom what they learn in staff 
development training. Table A-1 lists some of the advantages and disadvantages of 
formal classroom observation as an evaluation method. 

Table A-1 

Advantages and Disadvantages of Classroom Observation 



ADVANTAGES 



DISADVANTAGES 



• Observation can be hignly credible 
when seen as the report of what ac- 
tually took place presented by disin- 
terested outsider(s). 

• Observers provide a point of view 
different from that of people most 
closely ^:onnected with the program. 



Ihe presence ot observers may al- 
ter what takes place. 

• Time is needed to develop the in- 
strument and train observers. 

• Credible observers must be 
located. 

9 Time is needed to conduct suffi- 
cient numbers of observations. 

• Scheduling problems are common. 

• Observation data can be very dif- 
ficult to analyze and interpret. 
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Behavioral checklists. Behavioral checklists usually include a list of a few (no 
more than 10) behaviors that should (or should not) occur in the classroom. The 
task of the observer is simply to make a tally mark on the checklist each time he or 
she observes the behavior taking place. Sample behaviors include: 

• Teacher asks a question in LI. 

• Sf'ident asks a question in LI. 

• Student uses English to answer a question posed in English. 
9 Studenruses English to answer a question posed in LI. 

« Student uses LI to answer a question posed in LI. 

• Student uses LI to answer a question posed in English. 

• Teacher translates (into English) student's question posed in LI. 
9 Teacher reinforces student's English-language response using LI. 

As should be apparent from this listing, an almost endless variety of be- 
haviors could be included. Since observers cannot deal with more than 10, however, 
it is crucial that you identify the specific teacher and student behaviors that are 
thought to be most critical to the project's success. 

By reviewing the project's description, plans for staff development, lists of 
materials, etc., you should be able to identify teacher and student behaviors that are 
relevant to its instructional intent. You should then prepare an exhaustive list of 
such beha^/iors (perhaps including descri ons of behaviors that are counter to the 
project's instructional intent and should /. jur). Final selection of the behaviors 
to be included in the classroom observe* on instrument should be made by the 
project designer(s). 

Coded behavior records. Coded behavior records are more sophisticated than 
behavioral checklists in two respects: (a) ihey allow for the recording ^f more be- 
haviors, and (b) they make provision for the recording of sequences of behaviors. 
On the other hand, they have the disadvantage of being difficult to use and difficult 
to analyze. Unless you feel you really must know about sequences of behaviors we 
recommend that you not adopt coded behavior records. 
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Coded behav-or records usually consist of no more than a series of boxes. 
Behaviors are coded sequentially into the boxes using predetermined symbols. You 
might, for example, use the designation TQl to represent the teacher asking a ques- 
tion in LI, Then, if the student to whom the question was addressed answered in 
English, you might enter SAE into the next box. Figure A4 provides an example of 
a coded behavior record. 



TLB 


TQE 


SAE 


SQl 


TAE 









S 


= student 


T 


= teacher 


L 


= lecture 


Q 


= question 


A 


= answer 


E 


= English 


1 


= L1 



Figure A-1, Example of a coded behavior record. 

A three-element code is conimon, but more complex codes are possible. 
You might, for example, decide that it is important to know what kind of questions 
(fact versus inference, perhaps) the teacher asked in addition to the language in 
which the question was posed. 

When developing a coding system, remember that classroom events occur 
rapidly. Too complicated a coding system will overload the observer and result in 
lower quality data than a simpler -system. Make your coding system as simple as 
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possible and verify its feasibility by trying it out in actual classrooms. 

If possible, have more than one observer use the instrument to record data in 
the same classroom at the same time. If their coded records are significantly dif- 
ferent, the difficulty wjo^' by an excessively complicated coding system. 

Deleted report instruments. The delayed report technique is the least formal 
of the three classroom observation techniques discussed here, and the easie'Jt to 
implement. In it simplest form it involves no more than observing events and ac- 
tivities for a prescribed period of time (say, five minutes) and then witing down a 
description of what was observed. The recording instrument could be no more than 
a blank piece of paper. 

The reliability of observational data can be increased by using somewhat 
more structured delayed report instruments. For example, the observer might be in- 
structed to focus on two or three students who were working together and to record 
the percentage of a four-minute period they used English to discuss their task. 

The delayed report instrument might then include a question like the 
following: 

The small group of students used English about what percent of the 
time?- 

less than 25% between 50% and 75% 

^between 25% and 50% more than 75% 



Observation time periods should be short Observers should record their ob- 
servations immediately after the end of the time period and should be watching for 
a small number of events or activities. 

One advantage of the delayed report observation technique is that it enables 
the collecting of "richer" data than the two techniques described earlier. On the 
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other hand, richer data might require higher levels of inference to be made by the 
observer. 



Consider the following question: 



Hov/ would you describe the classroom climate? 

positive neutral/negative 

enthusiastic ^apathetic 

task-oriented ^disorganized 

all of the above all of the above 



Clearly the observer is being asked to make a subjective judgment, and dif- 
ferent observers would be likely to respond differently unless they had been 
thoroughly trained to interpret symptoms in the same way. To construct a useful 
delayed report instrument, project staff can meet and develop a list of questions 
about what goes on in project classrooms- The questions should then be tried out in 
actual classrooms to determine how accurately and reliably they can be answered- 

/ The training of observers who are subsequently called apon to make subjec- 

tive judgments necessarily involves identifying behavioral manifestations of such 
constnicts as "positive climate," An effective shortcut might therefore be to address 
your questions to the directly observable manifestations (e,g,, teacher frequently 
smiles and uses the word good) which conipnse the construct rather than to the con- 
struct itself. 



Selecting and Training Observers 

The essential qualifications of the observer(s) will depend upon the observa- 
tion method you choose. The more structured the method, the less knowledge of 
bilingual education the observer will require. In all cases, the observer should be 
objective and a non-stakeholder. 



After teaching the observer the observation procedure, you should give the 
observer an opportunity to try out what he or she has learned. This can be done 
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with a non-project class or with a videotape recording of a real class operation. 

During the "practice" observation or the first project class observation, the 
ratings of each observer should be compared with those of the trainer. The latter 
serve as the criteria. If a videotape is used, the consistency of the trainee's ratings 
can also be determined by comparing successive trials. Depending on the kinds of 
ratings generated, different coefficients can be computed to reflect the extent to 
which the trainee and the trainer agree (see Appendix B). 

You should identify and discuss the similarities and differences in the obser- 
vation data produced by each rater and the trainer. If the discrepancies are large, 
further training or a new observer may be needed. 

Growth or change estimates derived from observational data are likely to be 
biased if the collectors of such data become more proficient (possibly as a result of 
practice) or less proficient (possibly as a result of boredom or forgetting) over time. 
To minimize this source of potential bias, it is a good idea to provide some refresher 
training before each round of data collection. The need for consistency should be 
emphasized, and data collectors should be urged to follow documented procedures- 
even if they see room for improvement. 

The difficulties rissociated with collecting valid and useful observational data 
do not end with instrument development or observer training. An additional dif- 
ficulty is that students and teachers must get used to the presence of an observer 
before they will behave normally in front of one. You must be sure that this type of 
desensitizing has taken place before you begin "real" data collection. 

To be sure that the classroom events and activities you are observing are 
normal, you should observe each classroom on a number of different occasions and 
avoid conducting observations wLen atypical events such as parties or special per- 
formances are scheduled. Too small a number of observations will be likely to yield 
non-representative and unreliable data. Spreading your observations out over the 
entire school year will also enable you to detect changes that may suggest improved 
implementation as the project matures or worsening implcmenta.ion as teachers 
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"slip back" into earlier behavior patterns. 



If limited resources preclude your observing all project classrooms, you 
should select a sample that represents the full range of student, setting, and treat- 
ment characteristics. It will be more cost-effective to observe a sample of class- 
rooms on multiple occasions than all classrooms only once or twice. 

General Principles Applicable to Classroom Observation 

• Make the observation instrum. lU simple to use. 

a Conduct multiple observations, spread over the entire year. 
» Observe different features in different sessions. 

• Observe a representative sample of classrooms on multiple occasions 
rather than all classrooms only once or twice. 

© If possible, check intra- and/or interrater reliability. 

• Avoid conducting observations when atypical events are happening 

© If possible, avoid informing teachers the exact date when the observations 
will occur. 

• Try to conduct observations as unobtrusively as possible. 
Where ♦o Get Additional Information * 

The preceding discussion regarding classroom observation strategies and in- 
struments has necessarily been somewhat superficial. Although the important 
points have all been covered briefly, you would be well advised to do some addi- 
tional reading before actvially attempting to collect observational data. The follow- 
ing sources are recommended: 

Morris, L. L., & Fitz-Gibbon, C. T. (1978). How to measure program implementa- 
tion. Beverly Hills, CA: Sage Publications. 

Stallings, J. A. (1977). Learning to loot a handbook on classroom observation and 
teaching models. Belmont, CA: Wadsworth Publishing Company. 
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APPENDIX B 
INTERRATER RELIABILITY 



« 



B-1 

160 



A persistent problem with human observers is that they can look at the same 
things but perceive them differently. Two different observers for example, might 
describe the same classroom in totally different terms. 



The difference between these two descriptions is clearly not a function of the 
classroom itself. It results from different observer perceptions and is a classic ex- 
ample of low interrater reliability. If the two observers were asked to count the 
number of teacher-initiated positive and negative reinforcements of students' be- 
havior during a given time period, we would probably find more agreement between 
their responses. Reliability, or lack thereof, may be just as much a function of the 
rating task as it is of the rater. 

Usually, interrater reliability can be improved either by training the observer 
or, alternatively, by making the rating task more objective. All other things being 
equal, for example, you will obtain higher interrater reliabilities if the task is to 
count the frequency with which the teacher uses the word "good" than if it is to rate 
the supportiveness of the classroom climate. Even with extensive training, you may 
not be able to obtain acceptable levels of interrater reliability with the kind of high- 
inference tasks represented by describing the climate of a classroom. 

Quantifying Interrater Reliability 

In order to assess whether different observers would judge the same situation 
in the same ^ay, you need to measure their reliability. Once you have quantified in- 
terrater reliability, you know whether to proceed with the observations or whether 
additional work will be needed before you can use your observers in your 
evaluation. 

Different quantification techniques are required (and/o^ appropriate) for 
different rating tasks. For example, one oi the examples used above requires ob- 



Observer A 
Noisy, chaotic, and 
out-of-control 



Observer B 

Individualized, spontaneous, 
and creative 
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servers to categorize teacher behavior, while another requires them to rate class- 
room climate. For different types of tasks, different methods are appropriate for 
determining interrater reliabUity. 

Categorizing. Several techniques are available for assessing reliability in 
categorizing tasks. One of the simplest and best, however, is coefficient Kappa. 

Suppose that Observer A and Observer B independently rate 200 
teacher utterances as to whether they are positively reinforcing (P.R.), neutral (N), 
or negatively reinforcing (N.R.). Their ratings could be tabulated as shown in Table 
B-1 below. 



Table B-1 

Agreement/Disagreement Matrix of Observations 



Observer A 




P.R. 


N. 


N.R. 


Total 


P.R. 


66 


8 


6 


80 


Observer N. 


6 


35 


9 


50 


B N.R 


7 


14 


49 


70 


Total 


79 


47 


64 


200 



The number 66 in ^he upper left hand corner of the matrix shown in Table B- 
1 means that both observers agreed in categorizing 66 of the teacher utterances as 
positively reinforcing. The number 7 in the lower left hand corner means that 7 of 
the teacher utterances .ated as positively reinforcing by Observer A were judged to 
be negatively reinforcing by Observer B. 

The values in the main diagonal (the diagonal that runs from the upper left 
to the lower right of the matrix and, in this case, includes the numbers 66, 35, and 
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49) represent agreements between the two observers. The off-diagonal entries rep- 
resent disagreements. In calculating Kappa, we only need to be concerned with cells 
on the main diagonal - those representing agreements. 

Sometimes interrater reliability is incorrectly expressed simply as the percent 
of the total number of observations that are agreements. Agreements will occur by 
chance, however, even if the ratings of the two observers are totally unrelated. 
Kappa (appropriately^ corrects for chance agreements. 

The number of agreements that would be expected to occur by chance 
in any cell of the main diagonal is given by the corresponding row total multiplied by 
the corresponding column total and divided by the grand total. The chance expecta- 
tions for the middle cell of the matrix .shown in Table B-1 is thus (47 x 50) :-200 or 
11.75. It is because the actual number of agreements (35) is greater than the chance 
expectations that interrater reliability is greater than zero. 

The actual calculation of Kappa proceeds as follows: 

Step 1. Sum the actual number of agreements (the values in the main 
diagonal of the matrix). 

Step 2. calculate and sum the chance expectations for each cell 
in the main diagonal. 

Step 3. Subtract the result obtained in Step 2 from that obtained in Step 1. 

Siep 4. Subtract the result obtained in Step 2 from the total number of 
ratings. 

Step5. Divide the result obtained in Step 3 by the result 
obtained in Step 4. The "answer" is Kappa. 

For illustration purposes. Kappa is calculated as follows for the matrix shown 
in Table B-1. 
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Step 1. 66 + 35 + 49 = T^O. 

Step 2. (79 X 80) r 200 = 31.60 
(47 X 50) r 200 = 11.75 
(64x70)7 200 = 22.40 
65.75 



Step 3. 150 - 65.75 = 84.25 

Step 4. 200 - 65.75 = 134.25 

Step 5. 84.25 r 134.25 = .628 = Kappa 

The obtained value of Kappa in this example (.628) is lower than we would 
like. If the two observers agreed on all of their ratings, Kappa would be 1.00. Thus, 
there is substantial room for improvement and additional training of the observers 
should probably be undertaken. An alternative might be to have the observers tally 
the frequencies with which the words yes, good, no, and bad are used, since this 
rating task requires no inferences on the part of the observer. 

Ranking or rating. Kappa is an appropriate index of interrater reliability 
where entities, events, or behaviors are categorized. When numerical values are as- 
signed to entities, events, or behaviors, however, some other type of index must be 
used. Consider the following example in which 14 classrooms were rated by two in- 
dependent observers. 

In this type of situation, the index of interrater reliability to use is the correla- 
tion between the two sets of ratings. You should note, however, that there are 
several types of correlation coefficients. You may be able to compute a "regular" 
(product moment) correlation coefficient using a pocket calculator or personal 
computer. That type of coefficient, however, makes assumptions about the 
properties of the rating scale that will probably not apply to your data. For this 
reason, we recommend converting the ratings to ranks and calculating a 
rank-orer correlation coefficient. 
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Table B-2 
Ratings of Two Independent Observers 

Rating 



Classroom Observer A Observer B 



1 


6.0 


6.5 


2 


7.3 


6.8 


3 


2.2 


3.8 


4 


4.5 


5.0 


5 


9.7 


9.0 


6 


4.3 


8.7 


7 


8.0 


7.7 


8 


7.9 


7.9 


9 


7.1 


6.4 


10 


7.3 


6.4 


11 


6.6 


6.4 


12 


8.5 


7.7 


13 


8.1 


9.2 


14 


6.4 


7.0 



Table B-3 

Data Required for Calculating a Rank Order Correlation 



Observer A Observer B 



Classroom 


Rating 


Rank 


Rating 


Rank 


Rank Dif. 


RD2 


1 


6.0 


11 


6.5 


9 


2 


4 


2 


13 


6.5 


6.8 


8 


-1.5 


2.25 


3 


2.2 


14 


3.8 


14 


0 


0 


4 


45 


12 


5.0 


13 


-1 


1 


5 


9.7 


1 


9.0 


2 


-1 


1 


6 


43 


13 


8.7 


3 


10 


100 


7 


8.0 


4 


7.7 


5.5 


-1.5 


2.25 


8 


7.9 


5 


7.9 


4 


1 


1 


9 


7.1 


8 


6.4 


11 


-3 


-9 


10 


73 


6.5 


6.4 


11 


-4.5 


20.25 


11 


6.6 


9 


6.4 


11 


-2 


4 


12 


85 


2 


7.7 


5.5 


-3.5 


12.25 


13 


8.1 


3 


9.2 


1 


2 


4 


14 


6.4 


10 


7.0 


7 


3 


9 
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Table B-3 repeats the data from Table B-2 but includes the ranks of each 
observer's ratings, as well as the ratings themselves. It also Includes a column of dif- 
ferences between the two observers' ranks and another column in which those dif- 
ferences are squared. 

To calculate the kind of rank-order correlation coefficient we recommend, 
the following steps are required: 

Step L Subtract the rank orders of one observer from those of the 
other observer. 

Step 2. Square each of the rank differences obtained in Step L 
Step 3. Sum the squared rank differences. 

Step 4. Multiply the sum of the squared rank differences (from Step 3) 
by 6. 

Step 5. Square the number of entities rated/ranked. 

Step 6. Subtract 1 from the squared number of entities rated/ranked 
(from Step 5). 

Step? Multiply the results of Step 6 by the number of entities 
rated/ranked. 

Step 8. Divide the result of Step 4 by the result of Step 7. 

Step 9. Subtract the result of Step 8 from h The "answer" is the rank 
order correlation coefficient. 

Using the data from Table B-3, we would proceed as follows: 

Step 1. (The rank differences are included in Table B-3). 
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Step 2. 



(The squared rank differences are included in Table B-3. 



Step 3. 



The sum of the squared differences is 170. 



Step 4. 



6x170 = 1,020 



Step 5. 



14x14 = 196 



Step 6. 



196 - 1 = 195 



Step 1. 



195x14 = 2,730 



Steps. 



1,020 f2,730 = 374 



Step 9. 



1 - 31 A - .626 = rank-order correlation coefficient 



Yoa should note that the presence of tied ranks (e.g. ranks 6 and 7 were tied 
for Observer A and both were given the rank of 6.5; ranks 10, 11, and 12 were tied 
for Observer B and all were given the rank of 11) causes the calculated rank-order 
correlation coefficient to be a slight overestimate of the true value. 

There is a correction for tied ranks, but its effect is usually too small to worry 
about. For example, the correlation juit calculated (.626) would drop only to .624 if 
the correction for tied ranks were applied. 

Once again, the obtained interrater reliability coefficient is lower than would 
be desired. In this particular case, there is one sizeable difference between the 
rankings of the two observers (classroom 6 was ranked 3 by one observer and 13 by 
the other). One might wish to investigate this difference to see whether there was a 
clerical error or to find out why the observers had such different impressions. Nor- 
mally, one would hope to obtain interrater reliabilities of .80 or higher. 

The two procedures for calculating interrater reliability coefficients 
presented above were selected from a large sample of possible approaches because 



t 



of their relative computational ease arid freedom from strong assumptions. Some of 
the other approaches do have advantages, however, (e.g., the analysis of variance 
approach for assessing the interrater reliability of multiple raters). A more com- 
plete presentation of the topic is beyond the scope of the Users Guide but the 
interested reader is referred to any of tlie large number of statistics and evaluation 
design textbooks that deal with this issue. 
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Why is Functional Level Testing Necessary? 

Publishers of major achievement tests usually construct tests with several 
levels, each of which is designed to have suitable content and difficulty for children 
of specific ages or in specific grades. Publishers' recommendations about the level 
of test to use at a given grade level assume that both the curriculum and the 
achievement levels of the students to be tested are typical for that grade level. 
Their recommendations, however, \yill often be inappropriate for special popula- 
tions, such as LEP students or students in need of compensatory education. Figure 
C-1 shows why it is important to test at a level that is appropriate for your students, 
when you are testing for the purpose of evaluating project effectiveness. 

Suppose that you are considering two levels of a vocabulary test, A. and B, 
These tests will sample a student's vocabulary and predict from each student's per- 
formance the size of his or her vocabulary. In your project, you have students T 
through Z. The publisher recommends test B for students of this age. 

The lowest possible score on the level B test indicates a vocabulary of about 
800 words. Students T, U, and V will all obtain this score on test B, despite the fact 
t.iat their vocabularies range from about 25 words up to almost 2,000 words. The 
level B test is too hard for these students, and even if students T and U made good 
progress in vocabulary during the year, they could easily have the same score on a 
level B posttest that they had on the pretest. 

The highest possible score on level B indicates a vocabulary of about 1,500 
words. This is what student Z's score will show, although this student has a 
vocabulary of about 3,000 words. The test is too easy for this 'tudent. If he/she also 
took level B as a posttest, the score would not sh.jw any growth in vocabulary, be- 
cause student Z obtained the highest possible score on the pretest. 

If the level A test is used (one level below the recommended level) similar 
problems result. The test is still too difficult for student T, and it is too easy for stu- 
dents X, Y, and Z. None of these students will obtain scores that give a true picture 
of their vocabularies. 
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NUMBER OF WORDS IN STUDENT'S VOCABULARY 



50 



500 

Vocabulary 
Test "A" 



T 



5.000 



Vocabulary 
Tesf'B" 



20.000 



50.000 



100.000 




# 9 



9 <» 



T U V W X Y Z 



Figure C-1. Example of student scores on two vocabulary tests. 
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In our example, students U and V should be tested one level lower than the 
level recommenued by the publisher, and student Z should be tested one level 
higher. Student T should be tested two levels lower. In some caset, you may need 
to move more than one level away from the publishers' recommendation in order to 
test a student at his or her functional level. It is more important to test students at 
an appropriate level so as to obtain an accurate gauge of their performance than it 
is to stay close to the publisher's recommended level. 

How to Choose the Appropriate Test Level for Your Students 

If you already administered a standardized test to your students, Chapter 5 
contains rules of thumb you can use to determine whether the test you administered 
was at your student's functional level. You can also use scores from tests ad- 
ministered in the previous year to make similar judgments, keeping in mind that 
students' functional levels will increase over time. If you don't have access to scores 
from standardized tests for your students, there are other methods you may be able 
to use to determine whether a publisher's recommended test is at your students' 
functional level. 

• Locator tests. The publisher of the test you want to use may publish a 
short locator test. You can administer this test to your students to 
identify the most appropriate Icvd of the test to use. 

• Teacher judgments. A teacher familiar with the test you are consider- 
ing and with a student's classroom performance can often estimate the 
most appropriate test level. 

• Instructional materials. The level of instructional materials students 
are using can be used as an indication of their functional levels. 

0 Past classroom grades. If a student's classrc^m grades are very low or 
very high, this might indicate that the test recommended for his/her 
grade level might not be at his/her functional level. 
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How to Use Scores From Functional Level Testing in the Gap Reduction Model 

Using functional level tests with the gap-reduction model presents no 
problem. Simply use scale scores (sometimes called expanded standard scores) to 
perform all calculations. Scale scores are level-free. That is,, if your student 
achieved a scale score of 60 on Level B of a test, you can assume that he/she would 
have achieved that same score on Level C. Simply i member to use the raw-score- 
to-scale-score conversion table for the test level the students took to look up their 
scale scores. 

If you are using a "live" comparison group, the comparison group students 
may have b ;n tested in level. Ir. this case, you will convert their raw scores to scale 
scores using the in-level table. If you are using the test's norms as your comparison 
group, you will always use in-level scale scores. That is, if your project group is 
finish- g third grade, you will obtain your norm group scale scores from the table for 
the test level recommended for students in the spring of third grade. Again, you will 
obtain your project group's scale scores *rom the table for the test level they took. 

Summary 

Functional level testing is sometimes necessary in order to obtain a good in- 
dication of students' abilities. If project students are tested at a level other than that 
recommended by the test publisher, the gap-reduction model can be implemented 
vvithout any difficulty as long as the test publisher provides scale scores or expanded 
standard scores which can be used in gap-reduction calculations. 
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As stated in the text of this Users' Guide, translating tests is a difficult task 
that should not be taken lightly. There is a good chance that your efforts will be 
successful, however, if the test to be translated possesses the following 
characteristics: 

• short questions. 
© active voice. 

o specific rather than general terms. 

• no metaphors or colloquialisms. 

a no vague words (probably, frequently, sometimes). 

• no subjunctive mood» 

Tests for use in kindergarten and the early elementary grades are likely to have 
most or all of these characteristics and thus represent prime candidates for transla- 
tion. Tests designed for use in higher grades will provide more difficult challenges 
and, if alternatives other than undertaking translations are available, you would 
probably be well advised to adopt them. 

According to the literature, the most common error made by "amateur" trans- 
lators is that of being too literal. When word-for-word and construction-for- 
construction translations are made, the result is often stilted and ungrammatical in 
the "new" language. What is even worse, from a test perspective, is that word dif- 
ficulty, and hence, item difficulty, may be changed dramatically when the most ac- 
curate word equivalencies are used. 

It is, of course, essential that the translated material get the original ideas 
across. It is equally important, however, that it read as it would if composed by a 
native speaker and that the difficulty of the vocabulary used match that of the 
original. A translation that strives to meet these three goals will be far more 
successful than one that attempts to achieve exact literal correspondence. 

"Back translation" is a technique that is often used to assess the success of a 
translation. When a person unfamiliar vwth the original text back translates, the 
back translation should bear close resemblance to the original. If "the spirit is will- 



ing, but the flesh is weak," after translation and back translation, comes out "the 
whiskey is good, but the meat is awful," somethin^i is amiss in the translation process. 

The example just cited almost certainly aro: ^ from a too literal original trans- 
lation. Such a translation would, no doubt, fail to convey the meaning of the 
original text. In fact, one might speculate that the original translation had a mean- 
ing more like the back translation than like the original text. 

When there are significant discrepancies between the original text and a back 
translation, it may be helpful to paraphrase the original English until a more trans- 
latable (and back-translatable) version is found. One might, for example, para- 
phrase the original text to read, "people have good intentions, but lack the resolve to 
follow through on them," We suspect that a forward and back translation of this 
rephrasing would come closer to conveying the original meaning than the literal 
translation that back translated so badly. Of course, it is not always the original text 
that creates the p 'oblem. Sometimes it is the first translation, A second try may be 
much more successful, 

A related problem may arise when the translator has a different ethnic back- 
ground from the students who will be tested (or from the back translator). One 
would not wish to use a Cuban, for example, to translate a text to be used with stu- 
dents of Mexican heritage, as many words have quite different meanings for the two 
ethnic groups. Test questions could be phrased (accidentally) so that they would 
have one correct answer for one group but a different correct answer for the other 
group. 

What has been said above can be summarized in the form of seven specific 
recommendations: 

• Choose a translator who is fully bilingual and fluent in the particular 
dialect used by your target students. 
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o Impress upon your translator that you do not want a word-for-word 
translation. You want a translation that conveys the meaning of the 
orginal text, in a natural sounding and grammatically correct manner. 
Finally, you want the translation to have the same level of vocabulary 
difficulty as the original text. 

• Choose a back translator who has the same qualifications as your 
"forward" translator. 

• When discrepancies are found between the original text and the back 
translation, work with your origimJ translator to identify the problem. 
Generate new translations directly from the original text as well as 
from paraphrasings of that text. 

• Have the retranslaticnr back translated. 

• Continue the process until a translation is achieved that can be back 
translated successfully, i.e., so that the back translation conveys the 
same meaning as the original text. 

• Review the translations for naturalness and correctness of grammar in 
the new language as well as for comparability of vocabulary difficulty. 
Make final adjustments. 

All of these recommendations have been distilled from the relevant litera- 
ture. They are, however, highly distilled. For more detailed guidance, the reader is 
referred to the article. Constructing Matching Texts in Two Languages: The Applica- 
tion of Propositional Analysis by Valdes, Barera, and Cardinas, that appeared in the 
Fall 1984 issue of the NABE Journal. It, and the references it cites, should be 
helpful. 
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■ APPENDIX E 



MAJOR PUBLISHERS OF NATIONALLY STANDARDIZED 
ACHIEVEMENT TESTS 
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American Guidance Service, Inc. 
Publishers' Building 
Circle Pines, MN 55104 
(612) 786-4343 

CTB/McGrawHill 
Del Monte Research Park 
2500 Garden Road 
Monterey, CA 93940 
(800) 538-9547 
(800) 682-9222 (California) 
(800) 649-8400 (Alaska, Hawaii, 
and Foreign Countries) 



American Testronics, Inc. 
P. O. Box 2270 
Iowa City, lA 52244 
(800) 553-0300 

The Psychological Corporation 
555 Academic Court 
San Antonio, TX 
(512) 299-1061 



The Riverside Publishing Company 
8420 Bryn Mawr Avenue 
Chicago, IL 60631 
(800) 323-9540 

(312) 693-0040 (Alaska, Hawaii and Illinois) 



Science Research Associates 
155 North Wacker Drive 
Chicago, IL 60608 
(312) 984-2195 
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Test reliability is a complex topic that cannot be adequately covered here. 
Interested readers should consult one of the many psychometric textbooks that 
cover the topic. For our purposes, it is sufficient to note that highly reliable 
achievement tests yield scores (we will call them observed scores) that accurately 
reflect whatever real achievement differences e>jst among the individuals tested. 
Scores obtained from less reliable tests reflect these real differences less accurately. 

Reliability is quantified in terms of reliability coefficients that have a theoreti- 
cal range from 0 (totally unreliable) to 1 (peifectly reliable). Tests used in educa- 
tional research and evaluation, hov;cver, usually have reliability coefficients that fall 
somewhere in the range from .60 to .95. These coefficients are related to the 
proportion of the differences among observed scores that result from real di 
ferences -among the individuals tested. Observed-score differences that do not 
reflect real differences among the individuals tested result from measurement error. 

The classic psychometric model specifies that observed variance- (what we 
have been calling differences among observed scores) equals true uiance (what we 
have been calling real differences among the individuals tested) plus error variance 
(what we have been calling measurement error). Since variance is quantified as 
squared standard deviations, we have: 



2 _ 2 2 

^ Gbser\'ed ~ ^ true ^ error 



The amount of variance due to measurement error (error variance) is largely 
dependent on the characteristics of the test. It is thus relatively constant regardless 
of the particular group tested. On the other hand, the amount of variance due to 
real differences among the individuals tested (true variance) is clearly a function of 
how large those differences are. 

If groups are homogeneous, the proportion of observed variance that is true 
variance will be relatively small. For heterogeneous groups, it will be relatively 
large. Since the error variance will be the same for the two typos of groups, it fol- 
lows that the reliability of a test will be lower if homogeneous groups are tested than 
if heterogeneous groups are tested. 
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The preceding point is extremely important to remember sin'-e the. reliability 
of a test will be much lower when it is used with a bilingual project group tban it will 
be when used with a nationally representative norming group. Since test piu. Jshers 
calculate their reliability coefficients from the scores, of nationally repre5,e.ntative 
samples, the coefficients they cite in their technical manuals do not provide a good 
indication of how reliable a particular test will be for your project group. Before 
you begin making reliability comparisons among candidate insiruments, you should 
thus estimate what those reliabilities will be for your group. 

To illustrate how this estimation process is done, consider the following ex- 
ample. We have a test with a national-sample reliability of .91 and a national- 
sample (observed score) standard deviation of 22. 

Since we know that reliability equals s^,..u„ 

^ observed 

we can solve for s^jj.yg as follows: 

^ observed 
s^rue = (484) 
4ue = 440 

Error variance then becomes 484 - 440 = 44. 

Now, if we assume that our project group has only half the true variance of a 
nationally representative sample (a reasonable assumption), we have: 

s^obs = 440/2 + 44 

s^obs = 264 
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Then, the reliability of the test for our project group becomes: 



reliability = 220 ^ §3 
264 



While the reliability reduction from .91 to .83 is not particularly dramatic, 
suppose that we are also considering a test that had a national-sample reliability 0^ 
.75. The national sample would have the same true variance (440) as before, but the 
observed variance would now be 440 7 .75, or 586.7. Error variance would then be 
586.7 -440 or 146.7. 

Now if we add the true score variance of our group (220) to the test's error 
variance we get an observed variance of 366.7. The reliability of the instrument for 
our group then becomes 220 7 366.7 or .60. 

As you can see, the reliability of the less reliable test drops substantially 
more (.15) than the reliability of the more reliable test (.80) when used with a 
homogeneous treatment group. The reliability difference between the two instru- 
ments is also larger for the homogeneous treatment group (.23) than it was for the 
national samples (.16). 

At this point, we have not yet made clear how test reliability affects the ac- 
curacy of growth estimates. Estimating the reliability of the evaluation instruments 
for the specific group to be evaluated (as described above) is, however, a necessary 
first step. Once those reliability estimates have been derived, we can begin to assess 
their impact on the standard error of growth estimates. 

The standard error of a growth estimate is given by the following formula: 



"growth 




pre post 



2r 



pre-post 



S.E.X 



pre 



S.E- 



X 



post 
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where 

= the standard error of the pretest mean 

pre 

S-E-x = the standard deviation of the posttest mean 

post 

^pre-post " correlation between pre- and posttest scores 
Note that, 

ifS.E,5j equals S.E.^ then: 
pre p,:,st 

S.E. growth = ^2S.E.\- 2 rp^e.post ^ SB?^ 

and 

if r = .5, 

S.E. growth =\/s!e2^ = S-E.^ 

As can be seen, if tlie two stated conditions are met, the standard error of a 
growth estimate is the same as the standard error of the mean pre- or posttest score. 

For the sake of simplicity, we will proceed as if those two conditions are met. 
At the end of this discussion, we describe what happens if they are not met. 

The standard error of a mean is given by: 

F-5 

ERIC J 84 



As >ou may remember from the preceding discussion, s^^^ increases as 
reliability decreases. Thus, the standard error of the mean (and, consequently, the 
standard error of the growth estimate) also increases as test reliability decreases. 

Without going into the algebraic derivation, the per ncrease in the 
standard error that results from choosing a less reliable test (L.R.T.) over a more 
reliable test (M.R.T.) is given by: 

^^^l-MRT - >^/rel.LRT 
% increase in S.E. = 100 x 

Going back to our previous example, were we to select the instrument with a 
"local" reliability of .60 over the instrument with a "local" reliability of .83, we 
would increase the standard error of the mean by: 

100 X 

or 17.6%. 

We have constructed Table F-1 to illustrate the percentage increase in the 
standard error of the mean that will be associated with selecting tests with specific 
lower reliabilities over tests with specific higher reliabilities. This table should be 
useful to you in estimating the impact on the accuracy of your growth estimates of 
test selections you may contemplate. 

When using Table F-1, you should keep two things in mind: 

• the percent increase in standard error is constant regardless of the size of 
your evaluation sample, but the absolute size of the increment may be in- 
consequential if your sample is large. Anything less than about a twen- 
tieth of a national-sample standard deviation is probably not worth worry- 
ing about. 



r 



0 



The standaid error ol a growth estimate may be either larger or smaller 
that the standard errors of the means that go into its computation. If the 
pretest-posttest correlation is .5 and the standard errors of the pre- and 
posttest means are unequal, the standard error of growth is less than .5- 
but it will be smaller if the correlation exceeds .5 (except as offset by dif- 
ferences between the standard errors of the pre- an^ posttest means). 
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/ Table F-1 

Percent Increase in the Standard Error of the Mean When a Less Reliable Test is Selected 
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This appendix deals with the estimation of project-related growth which is 
defined as the project group's total growth from pretest to posttest minus the growth 
it would have experienced had there been no project. In a true, randomized ex- 
periment, the total growth of the control gro jp is taken as an unbiased estimate of 
the growth the project group would have experienced had there been no treatment. 
Then, the project group's total growth minus the control group's total growth yields 
our estimate of project-related growth (the project's impact). 

Given this formula, the accurate estimation of treatment effects rests on the 
accurate measurement of project-group growth and the accurate measurement (or 
estimation) of control-group growth. Since, in bilingual education, randomly equiv- 
alent control groups are precluded by the legislative mandate that the neediest stu- 
dents be served, we have no choice but to abandon the task of estimating treatment 
effects or to use quasi-experimental estimation techniques in lieu of direct 
measurement of control-group growth. 

In some bilingual education settings, quasi-experimental estimation tech- 
niques may be the logical choice. In other settings, however, it may simply be im- 
possible to implement a quasi-experimental design in such a way that its results 
would be credible. In such instances, abandoning the estimation of treatment ef- 
fects is the most prudent course of action. 

Three quasi-experimental designs appear to have technical merit for evaluat- 
in^ bilingual projects. Each of them, however, has implementation requirements 
that many school districts will be unable to meet. These districts will thus have to 
confine their evaluation efforts to producing growth estimates and assessing gap 
reduction.* 

Overviev*^ of the Designs 

The following paragraphs briefly summarize the three designs and 
their implementation requirements. If, after reading these summaries, you believe 
you may be able to implement one or more of the designs, you should go on to the 
more detailed descriptions presented later in this appendix. 
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The non-equivalent comparison group design. The first design to be dis- 
cussed is the non-equivalent comparison group design. Despite the impression 
created by its name, the validity of this design hinges on the treatment and com- 
parison groups being as nearly equivalent as possible. With substantially dissimilar 
groups, the assumptions underlying the design are not met and the statistical ad- 
justments that can be made to correct for differences between groups are likely to 
be unsuccessful. 

The primary difficulty associated with implementing this design is finding a 
nearly equivalent comparison group. A suitable group will almost certainly not exist 
within the same school as the treatment group. It may be possible to identify an ac- 
ceptably simi'-^r group in another school in the same district. More likely, however, 
is the possible existence of a suitable comparison group in some other district where 
program entry criteria are significantly different from those in your district. 

To illustrate, your district may have a small proportion of LEP students and 
may thus be able to serve Lau Category C students. Another district may have a 
much higher proportion of LEPs and may determine that it should devote all of its 
bilingual rv^.sources to Category A and B students. The Category C students in that 
district might constitute a reasonable comparison group-at least for the Category C 
students in your project. (Note: the comparability of the two groups might be even 
higher than it appears on the surface, since language proficiency tests tend to have 
quite low correlations with one another. This means that significant numbers of 
students identified as Lau Category C using one test would fall into Lau Categories 
B or D when tested wifh a different instrument.) 

Of course :Lj feasibility of implementing a non-equivalent comparison group 
design depends not only oi the existence of a suitable comparison group but on your 
ability to obtain the nece.ssary data (test scores, etc.) from its students. If you can do 
this, you should read the detailed implementation procedures presented below. If 
you cannot, you should consider one of the two remaining designs. 

The grade-cohort design. What we refer to here as the grade-cohort design is 
really a hybrid that borrows some aspect of time series designs and some aspects of 
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the \on-equivalent comparison group design. Basically, it involves comparing the 
scores of students who have been in a bilingual project for some period of time with 
pre-entry scores obtained from other students who entered the project at the same 
grade level at v/hich the project students were postlested. The difficult implementa- 
tion requirement for this model is simply having enough students entering the 
program at different grade levels (and presumably without prior bilingual education 
experience) to provide a stable estima*'^ of how the project students would be per- 
forming had they not been served by the project. An example may help to clarify 
this requirement. 

Consider a school that receives large numbers of LEP students at all grade 
levels (perhaps because there is a large immigrant population). These students are 
all tested and then assigned to the bilingual education project. Suppose that, over 
the course of a few years, pretreatment scores were obtained for some 100 entering 
third graders. 

Presumably there would also be a group of beginning third grade project 
participants-some of whom had been in the project one year, some two years, and 
some three years. By comparing the posttest scores of these children with the time- 
of-entry scores of the third-grade new arrivals, an estimate of project impact could 
be derived. 

Although, in practice, the grade-cohort design should not be implemented in 
quite CO simplistic a fashion, the example serves to illustrate its most significant 
features-including the need for a substantial amount of baseline data. 

If you believe that this design could be implemented in your district, you 
should read the detailed implementation procedures for it that are presented below* 
If you feel that it cannot be implemented, you should consider the other two designs 
summarized here. 

The regression-discontinuity design. Just as the validity of the uon- 
equivalent comparison group design depends on a high degree of similarity betwe m 
the project and comparison groups, the validity of the regression-discontinuity 
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design hinges .on the two groups having no overlap whatsoever on the pretest 
measure. All students scoring below some predetermined cuioff value on the 
pret<='st become members of the project group and all students scoring above it be- 
come members of the comparison group- 

In addition to being able to establish and stick to a pretest cutoff score for as- 
signing students to project and comparison groups, you must meet two additional 
requirements. The first of these factors is that the two groups must be drawn from a 
single population. The population might, for example, be students from homes 
where Spanish is the language spoken most of the time. Students from this popula- 
tion might then be assigned to project and comparison groups on the basis of their 
scores on an English language proficiency test. It would be inappropriate to imple- 
ment the model with a mixed population of Hispanic and Anglo students if most of 
the students below the cutoff were Hispanics and most of those above the cutoff 
were Anglos, 

The second requirement is that there must be a substantial number of stu- 
dents and a fairly wide range of pretest scores both above and below the cutoff. In 
theory, the model will work with small numbers and homogeneous sets of test 
scores. In practice, however, yuur estimate of project impact will be quite unstable 
under such circumstances. 

If the conditions just described can be met in your district, you should read 
the detailed implementation procedures for the regression-discontinuity design 
presented below. If they cannot be met, you should consider the other two designs 
summarized here. If none of these can be implemented, you may skip the 
remainder of the material presented in this appendix. 

Implementing tne Non-Equivalent Comparison Group Design 

One way to implement the non-equivalent comparison group design is to 
compare the two groups* posttest scores afiei they have been adjusted, by means of 
covariance analysis, for whatever differences existed between the groups at pretest 
time. Single or multiple covariates may be used but, in either case, the covariance 
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analysis should be "corrected" for the unreliability of the pretreatment measures 
since it can be assumed that true rather than obsen'ed pretest scores mediate post- 
test performance. 

The accuracy of the non-equivalent comparison group design hinges on the 
similarity of the two groups. If the two groups are identical at pretest time on all 
variables that correlate with posttest scores, then no statistical adjustment is 
required. If pretest scores are the only difference between the two groups, then 
using them as a single covariate provides an appropriate correction. But even if 
pretest scores are identical, the groups may differ on variables such as 
socioeconomic status, attitude toward school, academic aptitude and achievement 
motivation. To the extent that such variables are correlated with posttest scores, 
they must all be reflected in a "correct" statistical adjustment for initial differences 
between the groups. Any differences between the groups that remain unknown or 
unmeasured and that are therefore not included in the covariance analysis will 
result in a systematic misadjustment of posttest scores. 

Given this situation, one might be tempted to measure everything he/she 
could think of to look for variables that (a) correlated with posttest scores, (b) dis- 
criminated between groups, and (c) had low intercorrelations v/ith each other, as 
these are the three characteristics "good" covariates should have. Unfortunately, 
such "fishing" for significant relationships capitalizes on random error and thus 
results in a misadjustment for "real" between-group differences. The appropriate 
strategy is to consider the manner in which the two groups came mto existence and 
the kinds of between-group differences that could be expected to result from the 
group-formation process. If the groups came from different communities, for ex- 
ample, one would suspect that there might be differences in socioeconomic status, in 
time spent in this country, in parent educational level, and even in the home lan- 
guage environment. Since all of these variables could be expected to correlate with 
posttest scores, all of them should be regarded as potential covariates. With project- 
size groups, however, probably no more than three covariates should be chosen 
Additional covariates would be likely to add more measurement error than relevant 
tme variance to the composite. 
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If there are pretest differences between groups, pretest scores should cer- 
tainly be the first covariate chosen since these scores will correlate most highly with 
posttest scores. Choosing among the remaining alternatives might be as much a mat- 
ter of cost and convenience as anything else. Whether students participate in free or 
reduced-price lunch programs, for example, might be a readily availaPe piece of in- 
formation, and, as such, a logical choice for a socioeconomic status covariate. If 
surveys of the home language environment have been conducted, the resulting in- 
formation represents another "free" and potentially useful covariate. It might even 
be scaleable (in terms of, say, percent English spoken) as opposed to dichotomous 
(some English spoken vs. no English spoken). 

Once the covariates have been chosen, the analysis can proceed. Data 
analysis with this design is, however, a nontrivial task and may require not only the 
use of a computer and a standard analysis program, but modifications to that 
program as well. 

There are standard computer programs for analysis of covariance with mul- 
tiple covariates in the BMD, SPSS, and SAS packages. None of them has a provi- 
sion for introducing a reliability correction, however, and this calculation will have 
to be done either by hand or by modifying the existing program. The problem is fur- 
ther complicated by the fact that one or more of the covariates is likely to have an 
unknown reliability. 

LISREL and EQS are two computer programs that appear able to deal 
with all these complexities. They are difficult and expensive to run, however, and 
would probably require your working with someone who had prior experience with 
them. 

Although we are unable to offer a solution guaranteed to overcome the 
various difficulties inherent in the task of correctly adjusting posttest scores to com- 
pensate for initial differences between groups, it is probably useful to discuss the na- 
ture of the problem. In covariance analysis (leaving out several plicating 
details), a multiple correlation is calculated between the independent variables 
(covariates) and the dependent variable (posttest scores). This correlation is the 
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primary determiner of the proportion of the initial difference between project 
and comparison groups that should be used to adjust the observed difference be- 
tween mean posttest scores. 

For the sake of simplicity, consider a case in which pretest scores were the 
single covariate and the comparison group outscored the project group by 10 points 
on the pretest but by only 5 points on the posttest. Apparently the treatment had 
the effect of improving the performance of the project group relative to the com- 
parison group. But it is not appropriate to assume that the entire 10-point pretest 
difference would have carried over to the posttest had there been no project. Any 
part of the pretest difference that resulted from random error (unreliability) would 
not be expected to carry over. Another part of the difference which would not carry 
over to the posttest would be true variance that was unique to the pretest (not also 
measured by the posttest). 

If there were no measurement error and the pre-and posttests measured ex- 
actly the saniie characteristic, it would be appropriate to use the full 10-point pretest 
difference to adjust ihe posttest difference, thus yielding a project-impact estimate 
of plus 5 points (-5 + 10 = +5), Under other circumstances, we would not use the 
entire 10-point pretest difference, and the estimate of project impact would be cor- 
respondingly less than 5 points. 

The "regular" covariajice analysis procedure would be to multiply the pretest 
difference by the "slope of the pooled, within-group regression line" and adjust the 
posttest difference by the result. Again making some simplifying assumptions, we 
can take this slope as equal to the pretest-posttest correlation-say .8, The regular 
covariance adjustment procedure would be to multiply the pretest difference of 10 
points by this ,8 and add the result to the difference between the two groups' mean 
posttest scores- -5 +8 = +3, The covariance-adjusted estimate of project impact 
would thus be a gain of 3 points. 

This regular covariance adjustment is appropriate when the project and 
comparison groups are formed by randomly assigning students drawn from a single 
population. The assumption, under such circumstances, is that measurement error 
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is correlated with observed scores so that the higher of the two observed scores is 
relatively higher than it should be (compared to the true score for the group) and 
the lower score is relatively lower than it should be. The difference between the ob- 
served pretest scores of the two groups is thus somewhat larger than the difference 
between the corresponding true scores. Clearly, then, one should only adjust the 
posttest scores for that portion of the pretest difference that is due to the real dif- 
ferences between groups and not for that portion that is due to the measurement er- 
ror (unreliability) inherent in the pretest scores. 

When project and comparison groups are samples from different popula- 
tions, rather than the same one, there is no reason to assume that any part of the ob- 
served difference between mean pretest scores is the result of systematic differences 
in measurement error. Measurement error will be correlated with obseived scores 
within groups but not across them. Thus, the regular covariance adjustment will be 
systematically too low by an amount that is directly proportional to the amount of 
measurement error inherent in the pretest scores. 

To remedy this problem, the pre-post correlation needs to be adjusted up- 
wards to what it would be if the pretest scores were perfectly reliable. The formula 
for making this adjustment is quite simple. 



f = 



''xy 



XX 



where 

fxy = the estimated reliability-correlated pretest-posttest correlation. 

= the observed pretest-posttest correlation, 
^xx = reliability of the pretest for the groups tested. 
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If r^x in the above equation were J9 and r^^ were .80, then would be .90. 
We would then adjust the posttest difference by nine-tenths of the pretest dif- 
ference, or nine points: 

-5 + 9 = 4 

Our estimate of project impact is now +4 points. 

It is hoped that the preceding discussion will give you some feeling as to how 
reliability-corrected covariance analysis "works" in adjusting posttest means to com- 
pensate for pretreatment differences between groups. Unfortunately, it does not 
provide a model for actually conducting such an analyds since it is unlikely that all 
" of the simplifying assumptions we made would be met in any particular real-world 
situation. Without those simplifying assumptions, things get substantially more 
complicated. 

It is possible to do an analysis of covariance-even a reliability-corrected 
analysis of covariance-by hand. The computations are very cumbersome, however, 
and you will almost certainly want to make use of a computer-especially if you use 
multiple covariates (which we recommend). You will probably need to seek assis- 
tance in carrying out the analysis regardless of whether you modify standard 
covariance analysis programs or employ the more complex causal modeling 
programs (LISREL or EQS). 

After you have obtained your estimate of project impact we also recommend 
that you calculate 95% confidence limits around it. The standard error of that es- 
timate should be included in or easily derivable from the computer printout. Multi- 
plying that standard error by ± 1 95 will give you the upper and lower boundaries of 
, your 95% confidence interval. Values of 1 95 are provided in Table G-1. 
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Table G-1 

Values of 1 05 as a Function of Sample Size (N) 
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12.706 
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4.303 
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3.182 
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2.776 
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2.571 


7 


2.447 
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2.365 
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2.306 


10 


2.262 


11 


2.228 


12 


2.201 


13 


2.179 


14 


2.160 


15 


2.145 


16 


2.131 


17 


2.120 


18 


2.110 



19 


2.101 


20 


2.093 


21 


2.086 


22 


2.080 


23 


2.074 


24 


2.069 


25 


2.064 


26 


2.060 


27 


2.056 


28 


2.052 


29 


2.048 


30 


2.045 


40 


2.023 


50 


2.009 


60 


2.001 


100 


1.984 




1.960 



It was suggested earlier that it might be possible to find a more similar com- 
parison group outside of the project group's school or even district than within those 
confines. Doing so, however, introduces a threat to the validity of your estimate of 
project impact that has not yet been discussed. The problem arises because the 
"regular" school program is likely to differ between schools within a district. Such 
differences are likely to be even larger between schools in different districts. While 
it is reasonable to assume that students in a bilingual project in a particular school 
would be in the regular program in the same school if there were no bilingual 
project, it is clearly less reasonable to assume that they would be in the regular 
program in some other school-particularly if that other school were in another dis- 
trict. Still, equivalence of regular programs is the assumption that is implicit in the 
design when it is implemented with a comparison group drawn from another school. 

When the non-equivalent comparison group design is implemented with a 
comparison group drawn from a different school, it is essential that the credibility of 
that implicit assumption be investigated. If it can be verified that there is no dif- 
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ference between the program the comparison group students participated in an 1 
that which the project group students would have participated in had there been no 
bilingual project, the validity of the estimate of project impact is intact. If there are 
differences, they should be documented and their effects estimated. Tiie credibility 
of the derived estimate of project impact hinges on a convincing argument that it 
could not be due to differences between the project and comparison schools' regular 
programs. 

Implementing the Grade-Cohort Design 

As mentioned earlier, the grade-cohort design involves comparing data ob- 
tained from students who have been in a bilingual project for some time with data 
obtained from new entrants at the same grade level. The baseline data may be ob--^ 
tained over a period of several years, but the assumption is always that the baseline 
data represent the level of performance that would be exhibit<jd by program par- 
ticipants at the same grade level if they did not participate in the program. Although 
this assumption is basically the same as that employed by the two models described 
earlier in this appendix, it may, as we shall see, have quite a different meaning in the 
context of this design. 

Once the data have been obtained and appropriately stratified (see below), 
implementing the model is extremely simple. Subtracting the mean pre-eniry score 
of the comparison group from the mean posttest score of the treatment group yields 
an estimate of the project's impact. The standard error of the estimate can be 
readily determined (see any elementary statistics text), and confidence intervals can 
be constructed as described above for the nonequivalent comparison group design, 

What would be ideal from a methodological point of view would be to have a 
district in which there was an influx of new families each spring (or fall) so that 
there were new children entering school and the bilingual project in time for spring 
(or fall) testing. Only children who had no prior bilingual education experience 
would be considered for the evaluation sample. 
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A^ter a year in the project, the posttest scores of, for example, third graders 
who entered the project as second graders could be compared with the pre-entry 
scores of the children who entered a£ third graders. After two years in the project, 
the posttest scores of students who entered as first graders could be compared with 
the pre-entry scores of students who entered as third graders. Obviously the same 
general strategy would work for other grade levels and for longer "stays" in the 
project. 

To avoid the possibility of the kind of self-selection bias that might result if 
students who were still enrolled in school after one, two, or three years were sig- 
nificantly different from students who stayed a shorter time, pre-entry data should 
be "stratified". This stratification would involve sorting pre-entry scores according to 
the length of time students remained in the school district. Thus the posttest scores 
of students who had been in the project for three years would be compared with the 
pre-entry scores of students who remained in the district three years. Second-year 
posttest scores would be compared with the pre-entry scores of students who 
remained in the district at least two years, and so on. 

In the ideal scenario described above, all students would have entered school 
and the program at the same time and just before an assumed spring (or fall) test- 
ing. In a more realistic scenario, students would enter throughout the school year. 
Under these circumstances, pre-entry scores should have been collected at the time 
students entered the program. 

If there were sufficient numbers of students, one could ignore (for evaluation 
purposes only) those who entered the program at times other than the month(s) 
when posttesting was done. Since large enough numbers are unlikely, however, the 
best strategy is to calculate a regression equation relating pre-entry scores to grade 
level (measured to the tenth of a month). The resulting equation should then be 
used to predict the scores that individual students would have obtained had they all 
been tested at the same time. The time for which the prediction is desired is the 
time at which students already in the program are posttested. 
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The regression (prediction) equation will have the form 
Y=Y+ rgySyG - rgySyG 



the predicted test score 
the mean of the observed test scores 
the grade level for which the predicted test score is desired 
the mean of the grade-level "scores" 

the correlation between the test scores and the grade-level scores 
the standard deviation of the test scores 
the standard deviation of the grade-level scores. 

As noted earlier, it is important, when using the grade-cohort design, to com- 
pare the posttest scores of program participants with the pre-entry scores oi similar 
children. The design, in fact, is a variant of what is typically called the posttest-only, 
control group design. As such, it rests on the assumption that the project and com- 
parison groups are randomly equivalent~in other words, that they are both random 
samples from a single population. 

The assumption of random equivalence may be questionable simply because 
the comparison group is created from historical data. In addition, the comparison 
group rnay include students from more than one year. Any population shifts that 
occurred between the pre-entr>' testing of the comparison group and the posttesting 
of the project group (such as the influx of new refugees) would clearly invalidate the 
assumption of random equivalence. An equally serious problen^ would arise from 
any systematic attrition from the project group that was unmatched in the com- 
parison group. 

Attrition will probably occur in the treatment group and will probably be 
non-random in nature-in other words, the students who drop out will differ sys- 
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tematically from the students who remain in the project. Students who remain in 
the project group should thus not be compared to all students who had pre-entry 
scores. It is essential that the pre-entry scores of students who dropped out of the 
program between their entry and the first posttest be eliminated from the com- 
parison group before any of the calculations described above be undertaken. ^ 

Once the proper (i.e., with the scores of dropouts eliminated) pre-entry mean 
score has been calculated, it should be subtracted from the mean posttest score of 
the treatment group. (Mean time in the program should also be calculated and 
reported for both groups to provide an indication of this comparability.) The dif- 
ference between the two means is the estimate of project impact. Its statistical sig- 
nificance can be tested using the standard l-test for independent groups. In our par- 
ticular case, the formula would take the following form: 

Y" - Y 
^c 

t = 

where 

Yp = the mean posttest score of the project group 

= the mean of the pre-entry scores of the comparison group 
Sg = the standard deviation of the comparison group's pre-entry scores 
Sp = the standard devia ion of the project group's posttest sc Jres 
Nq = the number of students in the comparison group 
Np = the number of students in the project group 



1. If we are looking at program effects of two, three, or more years of participation, 
it is essential to follow the same procedure of eliminating the pre-entry scores of 
students who dropped out of the program during the interval of interest. 
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The denominator of the above equation is the standard error of the project- 
impact estimate. The 95% confidence interval can be calculated (as described 
above) by multiplying the standard error by ± 1 05. 

As suggested earlier, estimates of project impact derived from the grade- 
cohort design mc^ not be strictly comparable to estimates derived from the other 
two designs discussed in this chapter. In both of those two designs, expectations of 
how students would have performed without the project are derived from com- 
parison groups that are observed over the course of a year during which they at- 
tended school but did not participate in a bilingual project. In the grade-cohort 
design, the no-project performance expectation is derived from pre-entry scores of 
students whose prior educational experience is likely to be unknown and may even 
be non-existent. 

While the pre-entry scores of students who had been in school the previous 
year would be comparable to the posttest scores cf comparison-group students at 
the same grade level in a non-equivalent comparison group design, that com- 
parability would not hold for students entering school for the first time. A no- 
project expectation derived even in part from students who did not attend school the 
previous year would certainly be lower thaa one derived from school attendees. Es- 
timates of project impact would be correspondingly larger. 

Both types of no-project expectations ,re valid, but one tells us how project 
students would have performed if they had been in school but not in the project and 
the other tells us how project students would have performed had they neither at- 
tended school nor participated in the project. Evaluators using either the non- 
equivalent comparison group design or the regression-discontinuity design need not 
worry about this distinction. It may be important for evaluators using the grade- 
cohort design, however. Those evaluators should attempt to determine and report 
the proportion (if any) of the comparison-group students who did not attend school 
the year prior to their entry into the program. Then, if removal cf these students 
from the comparison group will not make the group too small, they should be 
removed (but counted as members of the project group the following year). Better 
still, separate project-impact estimates should be calculated using the no-project- 
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but-school expectation and the no-project-and-no-school expectation. 

Implementing the Regression-Discontinuity Design 

The regression-discontinuity design represents a special case of the non- 
equivalent comparison group design. Usually, the appropriate implementation of 
non-equivalent comparison group designs requires finding comparison groups that 
are as similar to the corresponding project groups as possible in all educationally 
relevant ways. In the regression-discontinuity design, the strategy is nearly the op- 
posite. A group of students is subdivided into project and comparison subgroups so 
that there is no overlap whatsoever between their pretest scores. A cutoff score is 
established, and all students on one side of it are assigned to one subgroup while all 
students on the other side are assigned to the other subgroup. 

One subgroup participates in the project while the other does not. Then both 
subgroups are posttested. Finally (in. the simplest analysis strategy), within- 
subgroup regression lines are calculated. The distance between their intercepts with 
the cutoff score represents the impact of the project. Figure G-1 illustrates the 
regression-discontinuity design in a situation where regression is linear and where 
the project has had a substantial impact. At the level of the Figure G-1 example, 
the regression-discontinuity design is clear and easy to interpret. Unfortunately, 
real data rarely look like the illustrations one finds in statistics books. And when 
one fits linear regression lines to real data, the design may produce "pseudo effects." 
Experience with Chapter 1 evaluations was sufficient to demonstrate that a variant 
of the model, perhaps because of problems related to its implementation, tended to 
yield negatively biased estimates of project impact. 

The design is particularly sensitive to test ceiling and floor effects, both of 
which produce curvilinear regression. Fitting linear regression lines to curvilinear 
data can produce either positive or negative pseudo effects, depending on where the 
cutoff score is located. To a large extent, however, the problem can be avoided by 
using curvilinear regression lines. 
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Figure G-1. The regression-discontinuity design (in its simplest 
form) with a substantial treatment effect. 



Before discussing strategies for data analysis, it is perhaps useful to review 
the conditions under which the regression-discontinuity design will "work." The ac- 
curacy of a project-impact estimate derived from the regression-discontinuity design 
depends on how "tightly" the data points define the regression lines. That tightness, 
in turn, is a direct function of the sample size and the within-group correlations be- 
tween pre-and posttest scores. Although, in theoiy, the design can be implemented 
with only a few students above and below the cutoff and a very low pre-post correla- 
tion, it would almost certainly be a waste of time and effort to do so. 
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recommendation is no more than a rule of thumb based on the probability that a 
"real" project effect will prove to be statistically significant, it is probably good ad- 
vice. If pre-post correlations are high, fewei than 30 students might suffice, but if 
correlations are low, 30 students might not be enough. With respect to conelations, 
.40 has been recommended as a minimum. Although, again, this proposed cutoff 
value is purely arbitrary, trying to find a statistically significant project effect in the 
presence of a correlation much below .40 will probably prove fruitless-especially if 
sample sizes are small. 

It is perhaps worth pointing out that correlations between sets of scores for 
any particular group of students depend to a large extent on the range of "talent" 
represented in the group. Two sets of scores will intercorrelate more highly in 
heterogeneous groups than in homogeneous groups. Ideally, then, one would like to 
have at least a moderate range of talent represented in both the above-cutoff and 
below-cutoff groups when implementing a regression-discontinuity design. The 
larger the range of uAent, the higher the pretest-posttest correlation will be. 

Another requirement, if the design is to work, is that a fixed cutoff score be 
established and that the assignment of students to groups be based strictly and en- 
tirely on that cutoff score. Breaking the rules is likely to introduce significant biases 
into effect-size estimates derived from the design. Analytic strategies have been 
developed for dealing with the problem of "fuz:^ cutoffs" but we recommend that 
you use a fixed cutoff if at all possible. 

Administrators and others may be reluctant to base participation on a fixed 
cutoff score on a single instrument. Fortunatel>, there is no need to do so. Multiple 
measures-including teacher judgments-can be used. The only requirement is that 
they be combined in such a way as to yield a single score. Once such a composite is 
developed, a cutoff score should be established for it. 

At this point it is important to note that assignment to a bilingual project is 
certain to be based on some kind of assessment of English language proficiency. 
The regression-discontinuity design is thus clearly usable to assess the project's ef- 
fectiveness in enhancing English language proficiency. Although it is less intuitively 
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The regression-cjiscontinuity design is thus clearly usable to assess the project* s ef- 
fectiveness in enhancing English language proficiency. Although it is less intuitively 
icnsible, the design can also be used to assess the effectiveness of the project's math, 
science, or social studies components (if the project has those components) even 
though the selection/pretest measure is not relevant to outcomes in those subject 
matter areas as long as the criterion of within-group correlations of .40 or larger is 
met. 

The final lequirement for valid implementation of the regression- 
discontinuity design is the most important and was alluded to earlier. The students 
above the cutorf score must be representatives of the same population as the stu- 
dents below it. If the majority of students below the cutoff belong to a particular 
ethnic minority group, the same proportion should hold for the group above the 
cutoff -otherwise there might be a discontinuity between the regression lines of the 
t^\'0 groups that has nothing whatsoever to do with project impact. 

As far as data analysis is concerned, the currently recommended procedure is 
to begin with a simple linear regression model such as that illustrated at the begin- 
ning of this discussion and then to reanalyze the data asing curvilinear regressions of 
successively higher orders. After each analysis, the regression lines are plotted and 
visually inspected. After a number of analysis iterations (say 10) the size of the 
project-effect estimate, the amount of variance explained by the model, and the 
residual mean square values should all be plotted against the order of the regression 
equation used. The evaluator must then use judgment to select the equation that 
both provides a good fit to the data and makes intuitive sense. Unfortunately, there 
15 no simple or mechanical way to select the best equation. 

Although the analytic process sounds intimidating, and, indeed, manual com- 
putations are out of the question, explicit guidelines are available for using two 
packaged computer programs (SPSS or MINITAB). Only slight modifications of 
these guidelines are required for other statistical packages (S ^ S, BMDP, Datatest). 
The interested evaluator is referred to Research Design for Program Evaluation: The 
Regression-Discontinuity Approach by W.M.IC Trochim (Sage Publications, 1984) for 
additional informatioa 
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The computer printout should provide you with a standard error of the 
effect-size estimate. This statistic enable you to compute a 95% confidence in- 
terval for your estimate following the same procedure described abo\'e for the non- 
equivalent comparison group design. 
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The basic gap-reduction calculations are performed using pre- and posttest 
scores, and all students in the evaluation sample must have both. Before any gap- 
reduction calculations are done, however, several preliminary steps may be 
required 

Because the regulations require a 12-month testing interval, the posttest for 
one year will often also serve as the pretest for the next year. Each year's posttest 
may also serve a third function-that of determining (or helping to determine) 
which, if any, students will be exited from the project. If any students are exited at 
the end of the year and their posttest scores were considered in the exiting decision 
these scores must be used in the subsequent year's evaluation even though the exited 
students did not participate in the project. Those scores are essential for computing 
the correction that must be made for the regression effect bias. 

If your project operated last year and served any students who were exited at 
the end of the year based at least partially on their posttest performance, you should 
calculate the regression-effect correction before you go any farther. The computa- 
tional procedures are presented in Appendix J. 

Note: No correction is required if this year's participants 
were separately pretested-that is, if last year's posttest scores 
did not also serve as this year's pretest scores. 

A second preliminary step that you may have to undertake involves estimat- 
ing the scores that would have been obtained by students who were pretested late or 
posttested early had they been tested at the regular testing times. These estimated 
regular-testing-time scores are derived through a process of extrapolation. The 
computational procedures are described in Appendix I. 

Note: You only have to do the score extrapolations described 
•n Appendix I if 10% or more of the students served by your 
project were either late enterers or early leavers (any students 
who participated in the project for less than 100 days need not 
be counted as served). 
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Once the preliminary steps have been completed, you can proceed with the 
gap-reduction calculations. To begin, you must compute the following statistics: 

• The project group's mean pretest score. 

6 The project group's median pretest score. 

o The comparison group's mean pretest score (if you are using normative 
data as your comparison group, the scale score corresponding to the 50th 
percentile). 

• The standard deviation of the comparison group's pretest scores (if you 
are using normative data as your comparison group, the scale score cor- 
responding to the 84th percentile minus the scale score corresponding to 
the 16th percentile divided by two). 

« 

® The project group's mean posttest score. 

• The project group's median posttest score. 

o The comparison group's mean posttest score. 

» The standard deviation of the comparison group's posttest scores. 

Once you have these statistics in hand, the next step is to decide whether to 
use the project group's mean or median pre- and/or posttest scores in future cal- 
culations. We recommend the folloudng decision criterion: if the mean exceeds the 
median by more than a fifth of a comparison group standard deviation, use the 
median-otherwise use the mean. 

Before proceeding-if the test you use has scale scores but some of your 
statistics are expressed in raw scores-you should convert the latter to the former. 
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Be sure to use the raw-to-scale-score conversion table that is appropriate for the test 
level you used. ^ 

The Pretest Gap is the comparison group's mean pretest score minus the 
project group's mean/median pretest score divided by the comparison group's 
pretest standard deviation. 

The Posttest Gap is the comparison group's mean posttest score minus the 
project group's mean/median posttest score divided by the comparison group's post- 
test standard deviation. 

The Gap Reduction is the pretest gap minus the posttest gap. 

The Comparison Group's (standardized) Growth is the comparison group's 
mean posttest score minus its mean pretest score divided by the square root of the 
average of its pre- and posttest squared standard deviations.^ 

The Project Group's (standardized) Growth is the comparison group's growth 
plus the gap reduction. 

The Relative Growth Index is the project group's growth minus the com- 
parison group's growth divided by the comparison group's growth and multiplied by 
100 (to convert it to a percentage). 



1. To convert raw-score standard deviations to scale-score standard deviations, find 
the scale score corresponding to the raw-score mean plus one (raw score) standard 
deviation and the scale score corresponding to the raw-score mean minus one raw- 
score standard deviation. Subtract the latter from the former and divide by two. 

2. First square the comparison group's pretest and postte;.. standard deviations. 
Add them together and divide by two. Then, take the square root of the result. 
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If you wish to prepare a graphic display of your results (see Figure H-1), plot 
the comparison group's pretest score as zero. Next, plot the project group's pretest 
score minus zero as the pretest gap. 

The next point to plot is the comparison group's pretest score, which is zero 
plus the comparison group's growth. Finally, plot the project group's posttest score 
as the comparison group's posttest score minus the posttest gap. 
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Figure H-1. Illustration of gap reduction. 
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If your project serves substantial numbers (more than 10%) of students who 
arrive after the pretest and/or leave before the posttest, you will need to obtain test 
scores from them in order to report represemative achievement data. Under these 
circumstances, you will administer pre- and/or posttests to them at times that may 
be several months away from the dates you tested the rest of your project students. 
The test scores for these late-arriving/early-leaving students will show growth based 
on less than a full school year of proje-t participation. In order to combine their 
scores with the scores of students who aid receive a full school year of project serv- 
ices, you will need to extrapolate the scores of the late-arriving/early-leaving stu- 
dents. After extrapolation, their scores will represent an estimate of how the stu- 
dents would have performed if they had received a full year of project service and 
had been tested at the normal testing times. In order for such an estimate to be 
valid, the students must have received project services for a long enough period that 
one could expect the services to have made a difference. A good rule of thumb is to 
include scores only for students who have participated in the project for at least 100 
school days. 

Early-Leaving Students 

The simplest case involves students who have "regular" pretest scores but 
who leave the project before the posttest would normally be administered. If 
forewarned of such students' departure dates, you can posttest them early. You 
then need to estimate what those students' scores would have been had they been 
posttested at the regular time. 

If you are on a spring-to-spring testing schedule, the estimation procedure is 
a straightforward linear extrapolation (explained below). Fall-to-fall (and possibly 
others) testing schedules, however may involve projections that span the summer 
months. Since students are not in school and presumably not in the project during 
the summer months, it is not reasonable to assume that they would grow at the same 
monthly rate as they do during the school year. V/e follow common practice in this 
matter and make the assumption that the per-month cognitive growth rate during 
the summer is one-third what it is during the school year. The three summer 
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months thus equal one school-year month and a full school year is made up of ten 
school-year months. 

Consider the following example. A project participant is scheduled to movt, 
out of the school district at the end of February. To avoid losing his data, he is post- 
tested on February 15th rather than the normal posttesting date of 30 April. Since 
the student was posttested 2.5 months eariy, his pretest-to-posttest growth reflects 
7.5 school-year months of "treatment" rather than the 10 school-year months. 

Our hypothetical student gained 9 test-score points (from 45 to 54) during his 
7.5 months of project participation. This average monthly growth was thus 9 :-7.5 or 
1.2 points. Assuming he would have continued to "grow at the same rate for the 
remaining 2.5 school-year months, he would have gained an additional 1.2 x 2.5 or 3 
points. We can then simply add that 3 points to his early postte^t score of 54. His 
estimated regular-testing-time score is thus 54 + 3 or 57. 

If, in our example, a fall-to-fall testing schedule had been adopted and our 
student was posttested in mid-May instead of September 30th. we would have had to 
extrapolate his score over 4.5 calendar months. In terms of school-year months, 
however, we would have the same 2.5 month period as in the eariier example. 

Late-Arriving Students 

Late arriving students present a more difficuh estimation problem. For one 
thing, it must be assumed that their pre-entry growth rate would have been less than 
their in-project growth rate. Thus we cannot simply make a linear projection back- 
wards to estimate what their pretest score would have been if they had been 
pretested at the regular time. 

For want of a better idea, we recommend assuming (as the computer 
program does) that a per-month growth during all months (summer and non- 
summer) prior to entry into the project was one-third of what was observed (per 
month) dunng project participation. To illustrate: Regular pre- and posttesting is 
accomplished at the end of April each year. Several new students who meet the 
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project*s entry requirements arrive in September, however, and are pretested on the 
15th-4.5 calendar months after the regular pretesting date. Assume that their 
average pretest score was 37 and their average posttest score was 52. They thus 
grew 15 points during 7.5 months of project participation-2 points per month. 

Had the student grown at one-third of their in-project rate during the 4.5 
months prior to entering the project, they would have grown 2 x 1/3 x 4.5 or a total 
of 3 points. Subtracting those 3 points from the late pretest score of 37 yields an es- 
timated regular-testing-time pretest score of 33. 

Unfortunately, our score-projection task is not over yet. Even though we 
have a real, regular-testing-time posttest score, we need to estimate what that score 
would have been had the late-entering students been in the project for the full year 
(10 school-year months). To continue with our example, we must multiply the 
average growth rate of two points per month by the 10 months in a full school year. 
This gives us 2 x 10 or 20 points estimated total growth over the school yeai. Adding 
those 20 points to the estimated regular-testing-time pretest score of 33 gives us 53 
as our estimate of what the late-entering students' mean posttest score would have 
been had they been in the project for the entire school year. 

All later pretest and early posttest scores should be replaced by the cor- 
responding regular-testing-time estimates before you begin your gap-reduction cal- 
culations. 
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Whenever the same test scores are used both to select students for participa- 
tion in a project and as their pretest measures, the so-called regression effect bias 
will be introduced. You are likely to encounter this problem in two different situa- 
tions. 

The first situation arises when students are initially selected to participate 
(perhaps based on low scores on a language proficienqr test). If those same scores 
are used as the students' pretest measures, they will be systematically distorted in a 
downward direction by the regression-effect bias. Measures of growth from pre- to 
posttest will, correspondingly, be spuriously inflated. 

If, after the students are selected, they are administered a separate pretest 
(an alternate form of the selection test or preferably, a different test altogether), you 
need not worry about the regression effect bias. Administering a separate pretest is, 
in fact, the procedure we reconmiend both because it obviates any need to adjust 
scores for statistical regression and because language proficiency tests are not well 
suited for measuring growth over time. 

The second situation where regression-effect biases will distort evaluation 
data is the one that is focused upon in Volume I of this Users' Guide. It arises when 
the posttest for one year of project participation also serves as the pretest for the 
subsequent year and some students are exited based, at least partially, on their post- 
test performance. In this situation, like the one described earlier, the 
posttest/pretest also serves as a selection test. Low scoring students are selected to 
continue in the project. Because of statistical regression their pretest scores will be 
lo ver than they should be and must be adjusted upwards to compensate for the 
regression-effect bias. 

If no students are exited, or if the exiting decision is made without knowledge 
of the students' scores on the posttest, then there is no "selection on the pretest'' 
and no significant amount of regression-effect bias. It would be nice if you could set 
your evaluation up so that you would not have to worry about the regression effect. 
Testing burden or other considerations may preclude that possibility, however. If 
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so, then you should use the following procedures to calculate the appropriate 
correction 

Step 1. Assemble last year's posttest scores (this year's selection and 
pretest scores) for all students who were pretested (including 
exited students and dropouts). 

Step 2. Calculate the mean and standard deviation of those scores. 

Step 3. Exclude the exited students and repeat Step 2 for the 
"restricted group." 

Step 4. Subtract the mean score of the restricted group (Step 3) from 
the mean score of the total group (Step 2). 

Step 5. Divide the standard deviation of the total group (Step 2) by the 
standard deviation of the restricted group (Step 3). 

Step 6. Identify that subset of students from Step 3 who were also post- 
tested at the regular time. 

Step 7. Calculate the pretest-posttest correlation for the students iden- 
tified in Step 6. 

Step 8. Enter Table J-1 with the ratio of the two group's standard 
deviations (from Step 5) and the pretest-posttest correlation 
calculated in Step 7. Read out the estimated correlation for 
the toial, unrestricted group. 

Step 9. Subtract the estimated total group correlation (from Step 8) 
from 1 A 
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Step 10. . Multiply the difference between the two group means (from 
Step 4) by the result of Step 9. The answer is the regression- 
effect correction. 

At this point we recommend that you add the regression-effect correction to 
the pretest scores of each of the students in the restricted group (from Step 3). The 
full correction is only appropriate, however, when posttest scores were the sole 
criterion for exiting students. If posttest scores and teacher judgments (and possibly 
still other factors) entered into the exiting decision, the full correction will be 
excessive. 

Under the circumstances just described, we recommend that you calculate 
separate gap reductions and RGI's using corrected and uncorrected scores and use 
the obtained values to "bracket" the correct values. 

One final note: Exiting decisions are likely to be based entirely on English 
language proficiency. If this is indeed the case, you do not have to worry about the 
regression effect biasing gap-reduction or RGI computations for LI proficiency or 
other academic subjects. 



J-4 

222 



Table J-1 

Table for Estimating the Total Group Pretest-Postlest Correlation 
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